Question

Find the minimum mean square error linear predictor of \(X_{n+1}\) given \(X_{n}\), \(X_{n-1}, \ldots, X_{0}\) in the following nonstationary linear model: \(\theta_{0}, \zeta_{1}, \zeta_{2}, \ldots\), and \(\varepsilon_{0}, \varepsilon_{1}, \ldots\) are all uncorrelated with zero means. The variances are \(E\left[\theta_{0}^{2}\right]=v_{0}^{2}\), \(E\left[\zeta_{k}^{2}\right]=v^{2}\), and \(E\left[\varepsilon_{k}^{2}\right]=\sigma^{2}\), where \(v^{2}=\alpha v_{0}^{2}, \alpha=v_{0}^{2}\left(v_{0}^{2}+\sigma^{2}\right) .\) Finally, \(X_{n}=\) \(\theta_{n}+\varepsilon_{n}\), where \(\theta_{n+1}=\theta_{n}+\zeta_{n+1}, n=0,1, \ldots\) (We interpret \(\left\\{X_{n}\right\\}\) as a noise distorted observation on the \(\theta\) process.) Answer: $$ \begin{aligned} &\hat{X}_{0}=0 \\ &\hat{X}_{k}=\alpha X_{k-1}+(1-\alpha) X_{k-1}, \text { for } k=1,2, \ldots \end{aligned} $$ where \(\alpha=v_{0}^{2} /\left(v_{0}^{2}+\sigma^{2}\right) .\)

Short answer

The minimum mean square error linear predictor for the given nonstationary linear model is: \[ \begin{aligned} &\hat{X}_{0} = 0 \\ &\hat{X}_{k} = \alpha X_{k-1} + (1-\alpha) X_{k-1}, \text{ for } k=1,2, \ldots \\ \end{aligned} \] where \(\alpha = \frac{v_0^2}{v_0^2 + \sigma^2}\).

Step-by-step solution

Define the variables and the model

We have the variables \(\theta_k, \zeta_k, \varepsilon_k\) with zero means and given variances, and the model is defined as follows: - \(X_n = \theta_n + \varepsilon_n\) - \(\theta_{n+1} = \theta_n + \zeta_{n+1}\) Our objective is to find the minimum mean square error (MMSE) linear predictor for \(X_{n+1}\) given the observed series \(X_n, X_{n-1}, \ldots, X_{0}\).

Derive the conditional expectation and variance

Since we want to minimize the mean square error between the true values and the predicted values, we need to derive the conditional expectation of \(X_{n+1}\) given \(X_n\), which is given by \(E[X_{n+1} | X_n]\), and the conditional variance \(Var(X_{n+1}|X_n)=E[(X_{n+1}-E[X_{n+1}|X_n])^2|X_n]\). First, we derive the conditional expectation: \[ \begin{aligned} E[X_{n+1}|X_n] &= E[\theta_{n+1} + \varepsilon_{n+1} | X_n] \\ &= E[\theta_n + \zeta_{n+1} + \varepsilon_{n+1} | X_n] \\ &= E[\theta_n | X_n] + E[\zeta_{n+1}] + E[\varepsilon_{n+1}] \\ &= E[\theta_n | X_n] \\ \end{aligned} \] Next, we derive the conditional variance: \[ \begin{aligned} Var(X_{n+1}|X_n) &= E[(X_{n+1} - E[X_{n+1}|X_n])^2 | X_n] \\ &= E[(X_{n+1} - \theta_n)^2 | X_n] \\ &= E[(\theta_n + \zeta_{n+1} + \varepsilon_{n+1} - \theta_n)^2 | X_n] \\ &= E[(\zeta_{n+1} + \varepsilon_{n+1})^2 | X_n] \\ &= E[\zeta_{n+1}^2] + E[\varepsilon_{n+1}^2] + 2E[\zeta_{n+1}\varepsilon_{n+1}] \\ &= v^2 + \sigma^2 \\ \end{aligned} \] Now that we have the conditional expectation and variance, we can proceed to derive the MMSE linear predictor.

Derive the MMSE linear predictor

As mentioned, we want to minimize the mean square error between the true values and the predicted values, which can be achieved by finding the MMSE linear predictor: \[ \begin{aligned} \hat{X}_{n+1} &= E[X_{n+1}|X_n] - \frac{Cov(X_{n+1}, X_n)}{Var(X_n)}(X_n - E[X_n]) \\ \end{aligned} \] As we know that \(E[X_{n+1}|X_n] = E[\theta_n | X_n]\) and \(Var(X_{n+1}|X_n) = v^2 + \sigma^2\), we can plug these into the formula above to get the MMSE linear predictor: \[ \begin{aligned} \hat{X}_{n+1} &= \alpha X_n + (1-\alpha)E[X_{n+1}|X_n] \\ \end{aligned} \] where \(\alpha = \frac{v_0^2}{v_0^2 + \sigma^2}\).

Present the final solution

The minimum mean square error linear predictor can be summarised as: \[ \begin{aligned} &\hat{X}_{0} = 0 \\ &\hat{X}_{k} = \alpha X_{k-1} + (1-\alpha) X_{k-1}, \text{ for } k=1,2, \ldots \\ \end{aligned} \] where \(\alpha = \frac{v_0^2}{v_0^2 + \sigma^2}\).