Question

Suppose \(X_{1}, X_{2}, \ldots, X_{n_{1}}\) is a random sample from a \(N(\theta, 1)\) distribution. Besides these \(n_{1}\) observable items, suppose there are \(n_{2}\) missing items, which we denote by \(Z_{1}, Z_{2}, \ldots, Z_{n_{2}} .\) Show that the first-step EM estimate is $$ \widehat{\theta}^{(1)}=\frac{n_{1} \bar{x}+n_{2} \hat{\theta}^{(0)}}{n} $$ where \(\widehat{\theta}^{(0)}\) is an initial estimate of \(\theta\) and \(n=n_{1}+n_{2}\). Note that if \(\hat{\theta}^{(0)}=\bar{x}\), then \(\widehat{\theta}^{(k)}=\bar{x}\) for all \(k\)

Short answer

\(\widehat{\theta}^{(1)}=\frac{n_{1} \bar{x}+n_{2} \hat{\theta}^{(0)}}{n}\) is the first-step Expectation Maximization estimate from a random sample of a normal distribution with missing data. Further, if the initial estimate equals to the sample mean, the EM estimate remains the same for all iterations.

Step-by-step solution

Identify Observations and Initial Guess

For the given scenario, we are given \(X_{1}, X_{2}, \ldots, X_{n_{1}}\) as random samples from a normal distribution with mean \(\theta\), and unit variance. Also \(Z_{1}, Z_{2}, \ldots, Z_{n_{2}}\) are missing items. We also have an initial estimate \(\hat{\theta}^{(0)}\).

Formulate the EM Estimate

Here in this step, we formulate the EM estimate. The Expectation-Maximization algorithm's essence is to use the observed data to estimate the missing data, and then use this completed data to estimate the parameters. So in our case, we're going to estimate \(\theta\) using the observed data (which is \(n_{1} \bar{x}\)) and our initial estimate of the missing data (which is \(n_{2} \hat{\theta}^{(0)}\)). This gives us the updated estimate of \(\theta\) which is \(\widehat{\theta}^{(1)}=\frac{n_{1} \bar{x}+n_{2} \hat{\theta}^{(0)}}{n}\).

Note the Peculiarity

Note that if the initial guess is the sample mean, i.e., \(\hat{\theta}^{(0)}=\bar{x}\), then the estimate remains the same for all steps of the iteration, i.e., \(\widehat{\theta}^{(k)}=\bar{x}\) for all \(k\). This remains valid because if the initial estimate was the sample mean, then the inclusion of the missing data (which are estimated to have the same mean) does not change the overall mean.