Question

Let \({\bf{f}}\left( {{{\bf{x}}_{{\bf{1}}\,}}{\bf{,}}{{\bf{x}}_{{\bf{2}}\,}}} \right)\) be a joint p.d.f. Suppose that \(\left( {{{\bf{x}}_{{\bf{1}}\,}}^{\left( {\bf{i}} \right)}{\bf{,}}{{\bf{x}}_{{\bf{2}}\,}}^{\left( {\bf{i}} \right)}} \right)\)has the joint p.d.f. Let \(\left( {{{\bf{x}}_{{\bf{1}}\,}}^{\left( {{\bf{i + 1}}} \right)}{\bf{,}}{{\bf{x}}_{{\bf{2}}\,}}^{\left( {{\bf{i + 1}}} \right)}} \right)\)be the result of applying steps \(2\,\,and\,\,3\) of the Gibbs sampling algorithm on-page \({\bf{824}}\). Prove that \(\left( {{{\bf{x}}_{{\bf{1}}\,}}^{\left( {{\bf{i + 1}}} \right)}{\bf{,}}{{\bf{x}}_{{\bf{2}}\,}}^{\left( {\bf{i}} \right)}} \right)\) and \(\left( {{{\bf{x}}_{{\bf{1}}\,}}^{\left( {{\bf{i + 1}}} \right)}{\bf{,}}{{\bf{x}}_{{\bf{2}}\,}}^{\left( {{\bf{i + 1}}} \right)}} \right)\)also have the joint p.d.f. f.

Short answer

The Gibbs sampling algorithm.

\(\left( {1.} \right)\,\)Pick starting values \({x_2}^{\left( 0 \right)}\) for \(\,{x_2}\) , and let \(\,\,i = 0\,\)

\(\left( {2.} \right)\,\)let be a simulated value from the conditional distribution \(\,{x_1}\)given that \(\,\,{X_1} = {x_2}^{\left( i \right)}\)

\(\left( {3.} \right)\,\)Let\(\,{x_2}^{\left( {i + 1} \right)\,\,}\,\) be a simulated value from the conditional distribution \(\,{x_2}\) given that \(\,{X_1} = {x_1}^{\left( {i + 1} \right)}\)

Use the Gibbs Sampling Algorithm

Step-by-step solution

Definition of Gibbs Sampling Algorithm

Gibbs Sampling is a Monte Carlo Markov Chain method for estimating complex joint distributions that draw an instance from the distribution of each variable iteratively based on the current values of the other variables.

The Gibbs Sampling Algorithm:

The steps of the algorithm are

\(\left( {1.} \right)\,\)Pick starting values \({x_2}^{\left( 0 \right)}\) for \(\,{x_2}\) , and let \(\,\,i = 0\,\)

\(\left( {2.} \right)\,\)let be a simulated value from the conditional distribution \(\,{x_1}\)given that \(\,\,{X_1} = {x_2}^{\left( i \right)}\)

\(\left( {3.} \right)\,\)Let\(\,{x_2}^{\left( {i + 1} \right)\,\,}\,\) be a simulated value from the conditional distribution \(\,{x_2}\) given that \(\,{X_1} = {x_1}^{\left( {i + 1} \right)}\)

\(\left( {4.} \right)\)Repeat steps \(\,2.\,\,and\,3.\)\(\,i\) where\(\,i + 1\)

Let \(f\left( {{x_1},{x_2}} \right)\) be the joint p.d.f. of \(\,\,\left( {{x_1}^{\left( i \right)},{x_2}^{\left( i \right)}} \right)\)The conditional distribution of\(\,{x_1}\) given that \(\,{X_2} = {x_2}^{\left( {i + 1} \right)}\), denoted \({f_1}\)with is

\({g_1}\left( {{x_1}\mid {x_2}} \right) = \frac{{\left( {{x_1},{x_2}} \right)}}{{{f_2}\left( {{x_2}} \right)\,\,\,\,}}\)

Marginal probability density function

In the case of a pair of random variables (X, Y), the density function of random variable X (or Y) considered alone is known as the marginal density function.

The \({f_2}\,\)is the marginal probability density function of \(\,\,{x_2}^{\left( i \right)}\).

Step 2 is \(\,{x_1}^{\left( {i + 1} \right)}\,\)a simulated value from the conditional distribution of \({x_1}\,\)given that \({X_2} = {x_2}^{\left( i \right)}\,\), which implies that the joint p.d.f. of \(\,\left( {{x_1}^{\left( {i + 1} \right)},{x_2}^{\left( i \right)}} \right)\) is the product of \(\,{g_{1\,\,}}\)and \({g_{2\,\,}}\)

\({g_1}\left( {{x_1}\mid {x_2}} \right)\,{f_2}\left( {{x_2}} \right) = f\left( {{x_1},{x_2}} \right).\)

Next, since it is also the joint p.d.f. of \(\,\,\,\left( {{x_1}^{\left( i \right)},{x_2}^{\left( i \right)}} \right)\), the marginal distribution must be the same as the marginal distribution \({x_1}^{\left( {i + 1} \right)}\)because integrating overall \(\,{x_2}\,\,\)gives the same function.

Similarly, in the algorithm step\({x_2}^{\left( {i + 1} \right)}\) is a simulated value from the conditional distribution of \({x_2}\,\)given that\({X_1} = {x_1}^{\left( {i + 1} \right)}\). Analogously, it must be that the marginal distribution of\(\,{x_2}^{\left( i \right)}\,\) must be the same as the marginal distribution of\(\,{x_2}^{\left( {i + 1} \right)\,\,\,}\), which implies that\(\left( {{x_1}^{\left( {i + 1} \right)},{x_2}^{\left( {i + 1} \right)}} \right)\,\,\) it must have the same joint distribution as\(\,\,\,\left( {{x_1}^{\left( i \right)},{x_2}^{\left( i \right)}} \right)\)

Hence,

Use the Gibbs Sampling Algorithm.