Question

For the Markov chain with states \(1,2,3,4\) whose transition probability matrix \(\mathbf{P}\) is as specified below find \(f_{i 3}\) and \(s_{i 3}\) for \(i=1,2,3\). $$ \mathbf{P}=\left[\begin{array}{llll} 0.4 & 0.2 & 0.1 & 0.3 \\ 0.1 & 0.5 & 0.2 & 0.2 \\ 0.3 & 0.4 & 0.2 & 0.1 \\ 0 & 0 & 0 & 1 \end{array}\right] $$

Short answer

In summary, to find the first passage probabilities \(f_{i3}\) and mean first passage times \(s_{i3}\) for \(i=1,2,3\) with given transition probability matrix \(\mathbf{P}\), we utilize the recursive formula for first passage probabilities and the definition of mean first passage time. Calculation involves iterating over the recursions until a desired level of precision is obtained.

Step-by-step solution

Calculate the first passage probabilities, \(f_{i3}\)

The first passage probabilities \(f_{ij}\) are defined as the probability that the Markov chain will first visit state \(j\) from state \(i\). They can be calculated recursively using the following formula: $$ f_{ij}^{(n)} = \sum_{k \neq j} p_{ik} f_{kj}^{(n-1)}, \quad n \geq 1, $$ where \(f_{ij}^{(n)}\) denotes the probability that the process first enters state \(j\) at step \(n\) and \(p_{ik}\) is the transition probability from state \(i\) to state \(k\). For \(i=1,2,3\), we will compute \(f_{i3}\) using the formula above: 1. When \(i = 1\), \(f_{13}^{(n)} = 0.1\cdot f_{33}^{(n-1)}+0.2\cdot f_{23}^{(n-1)}+0.4\cdot f_{13}^{(n-1)}\). 2. When \(i = 2\), \(f_{23}^{(n)} = 0.2\cdot f_{33}^{(n-1)}+0.5\cdot f_{23}^{(n-1)}+0.1\cdot f_{13}^{(n-1)}\). 3. When \(i = 3\), \(f_{33}^{(n)} = 0.2\cdot f_{33}^{(n-1)}+0.4\cdot f_{23}^{(n-1)}+0.3\cdot f_{13}^{(n-1)}\). With these recursions, we can calculate the first passage probabilities until the desired convergence is obtained.

Calculate the mean first passage times, \(s_{i3}\)

The mean first passage time, \(s_{ij}\), is the expected time it takes for the Markov chain to reach state \(j\) for the first time from state \(i\). It can be calculated using the first passage probabilities, \(f_{ij}^{(n)}\), as follows: $$ s_{ij} = \sum_{n=1}^{\infty} n \cdot f_{ij}^{(n)}. $$ Using the previously calculated first passage probabilities, \(f_{ij}^{(n)}\), for \(i=1,2,3\), we can now compute the mean first passage times, \(s_{i3}\), by summing the series for each \(i\). Remember that, in general, we must use numerical methods to compute both first passage probabilities and mean first passage times. By iterating over the recursions, we can obtain these values to a desired level of precision.