Question

To prove Equation ( \(7.24)\), define the following notation: \(X_{i}^{j} \equiv\) time spent in state \(i\) on the \(j\) th visit to this state; \(N_{i}(m) \equiv\) number of visits to state \(i\) in the first \(m\) transitions In terms of this notation, write expressions for (a) the amount of time during the first \(m\) transitions that the process is in state \(i ;\) (b) the proportion of time during the first \(m\) transitions that the process is in state \(i\) Argue that, with probability 1 , (c) \(\sum_{j=1}^{N_{i}(m)} \frac{X_{i}^{j}}{N_{i}(m)} \rightarrow \mu_{i}\) as \(m\) (d) \(\mathrm{N}_{i}(m) / m \rightarrow \pi_{i} \quad\) as \(m \rightarrow \infty\). (e) Combine parts (a), (b), (c), and (d) to prove Equation (7.24).

Short answer

In summary, we have found expressions for the time spent in state i and the proportion of time spent in state i during the first m transitions. We have also shown that the limits of the average time spent in state i converge to μ_i and the ratio of the number of visits to state i converges to π_i, both with probability 1. By combining these results, we have proved the equation (7.24): \[ \lim_{m\to\infty} \frac{\sum_{j=1}^{N_{i}(m)} X_{i}^{j}}{m} = \mu_{i} \cdot \pi_{i} \]

Step-by-step solution

(a) Time spent in state i during m transitions

To find an expression for the amount of time spent in state i during the first m transitions, we need to add the time spent in state i for each of the N_i(m) visits. The total time spent in state i during the first m transitions can be given by the sum: \[ \sum_{j=1}^{N_{i}(m)} X_{i}^{j} \]

(b) Proportion of time spent in state i during m transitions

To find the proportion of time spent in state i during the first m transitions, we need to divide the time spent in state i by the total time (m). The proportion of time spent in state i during the first m transitions can be given by: \[ \frac{\sum_{j=1}^{N_{i}(m)} X_{i}^{j}}{m} \]

(c) Limit of the average time in state i

We need to show that, with probability 1, the limit of the average time spent in state i converges to μ_i as m goes to infinity. This can be given by: \[ \lim_{m\to\infty} \sum_{j=1}^{N_{i}(m)} \frac{X_{i}^{j}}{N_{i}(m)} = \mu_{i} \] This holds because, as m increases and the process spends more time in state i, the average time spent in state i converges to the expected time spent in state i, which is μ_i.

(d) Limit of the ratio of visits to state i

We need to show that, with probability 1, the limit of the ratio of the number of visits to state i converges to π_i as m goes to infinity. This can be given by: \[ \lim_{m\to\infty} \frac{N_{i}(m)}{m} = \pi_{i} \] This limit shows that, as the total number of transitions m increases (goes to infinity), the proportion of visits to state i converges to its stationary distribution, which is π_i.

(e) Proving the equation (7.24)

Combining parts (a), (b), (c), and (d), we can write the desired equation (7.24) as: \[ \lim_{m\to\infty} \frac{\sum_{j=1}^{N_{i}(m)} X_{i}^{j}}{m} = \lim_{m\to\infty} \mu_{i} \cdot \frac{N_{i}(m)}{m} = \mu_{i} \cdot \pi_{i} \] This equation shows that as m goes to infinity, the proportion of time spent in state i during the first m transitions converges to the product of the expected time spent in state i (μ_i) and its stationary distribution (π_i), ultimately proving the equation (7.24).