Question

Suppose that we want to predict the value of a random variable \(X\) by using one of the predictors \(Y_{1}, \ldots, Y_{n}\), each of which satisfies \(E\left[Y_{i} \mid X\right]=X .\) Show that the predictor \(Y_{i}\) that minimizes \(E\left[\left(Y_{i}-X\right)^{2}\right]\) is the one whose variance is smallest. Hint: Compute \(\operatorname{Var}\left(Y_{i}\right)\) by using the conditional variance formula.

Short answer

To minimize the expression \(E\left[\left(Y_{i}-X\right)^{2}\right]\), we need to find the predictor \(Y_i\) with the smallest variance. Using the conditional variance formula, we can rewrite the expression as \(\operatorname{Var}(Y_{i}) - \operatorname{Var}(X)\). Since \(\operatorname{Var}(X)\) does not depend on \(Y_i\), minimizing \(E\left[\left(Y_{i}-X\right)^{2}\right]\) is equivalent to minimizing \(\operatorname{Var}(Y_{i})\). Therefore, the predictor \(Y_i\) with the smallest variance is the one that minimizes \(E\left[\left(Y_{i}-X\right)^{2}\right]\).

Step-by-step solution

Write down the expression we need to minimize

We are given that \(E\left[Y_{i} \mid X\right]=X .\) We need to minimize the following expression: \[ E\left[\left(Y_{i}-X\right)^{2}\right] \]

Use the definition of variance

Recall the definition of variance: \[ \operatorname{Var}(Y_{i}) = E\left[(Y_i - E[Y_i])^2\right] \] Since \(E\left[Y_{i} \mid X\right] = X\), we have \(E[Y_i] = E[X]\). Thus, the variance of \(Y_i\) is: \[ \operatorname{Var}(Y_{i}) = E\left[(Y_i - X)^2\right] \]

Use the conditional variance formula

The conditional variance formula states that: \[ \operatorname{Var}(Y_{i}) = E\left[ \operatorname{Var}(Y_i \mid X)\right] + \operatorname{Var}\left( E[Y_i \mid X]\right) \] We have \(\operatorname{Var}(Y_i \mid X) = E\left[(Y_i - X)^2\right]\). Now, we need to find an expression for \(\operatorname{Var}\left( E[Y_i \mid X]\right)\). Since \(E[Y_i \mid X] = X\), we have: \[ \operatorname{Var}\left( E[Y_i \mid X]\right) = \operatorname{Var}(X) \]

Rearrange and discuss the relationship

Now, we have the following relationship: \[ E\left[\left(Y_{i}-X\right)^{2}\right] = \operatorname{Var}(Y_{i}) - \operatorname{Var}(X) \] The term \(\operatorname{Var}(X)\) does not depend on the choice of the predictor \(Y_i\). Thus, minimizing \(E\left[\left(Y_{i}-X\right)^{2}\right]\) is equivalent to minimizing \(\operatorname{Var}(Y_{i})\). Therefore, the predictor \(Y_i\) with the smallest variance is the one that minimizes \(E\left[\left(Y_{i}-X\right)^{2}\right]\).

Key concepts

Conditional Expectation

The concept of conditional expectation is a fundamental building block in the understanding of probability models. It refers to the expected value of a random variable given that another random variable takes on a specific value. This is essential for understanding how the average or expectation alters under certain conditions. In the given exercise, the conditional expectation is denoted as \( E[Y_i | X] \), representing the expected value of \( Y_i \) given \( X \). The condition \( E[Y_i | X] = X \) sets up the framework for predictor selection, as it suggests that on average, the predictors \( Y_i \) are equal to the value of \( X \) they are trying to predict.
When dealing with conditional expectation in the context of selecting a predictor for a random variable, it serves as the cornerstone of analysis, guiding the subsequent steps in the process for prediction accuracy. By selecting a predictor whose conditional expectation matches the value of \( X \) itself, we aim to ensure that, on average, the predictor aligns well with the random variable it predicts.

Variance of Random Variables

Variance is a measure of dispersion, indicating how much a set of random numbers diverges from the mean of the set. In probability theory, the variance of a random variable tells us how spread out the values are around the expected value. The formal definition is \( Var(X) = E[(X - E[X])^2] \), where \( E[X] \) is the expected value of the random variable \( X \).

• The variance captures the variability of a random variable.
• Low variance indicates that values cluster close to the mean.
• High variance signifies that values are spread out over a wider range.

In the exercise provided, we analyze the variance of the predictors \( Y_i \) in the quest to find the one that could predict \( X \) most accurately. The step of computing the variance of each predictor is crucial because it essentially measures their reliability; the lower the variance, the more consistent the predictor. By aiming to select a predictor with the lowest variance, we limit the unpredictability in the predictions made.

Conditional Variance Formula

Expanding upon the variance concept, the conditional variance formula adds another layer by considering the variance of a random variable within given constraints or conditions. The formula is represented as \( Var(Y|X) = E[Var(Y|X)] + Var(E[Y|X]) \), which combines the expected variance given \( X \) and the variance of the expectation given \( X \).
This formula is essential when working with datasets where additional information can refine our uncertainty about a variable's behavior. It distinctly separates the embedded variability from that which is attributed to the conditional expectation itself. In the exercise, the formula assists in deconstructing the variance of the predictor \( Y_i \) into a portion that's conditional on \( X \) and another that deals with the variability in \( X \). Such decomposition is key to understanding how predictor selection can be fine-tuned by acknowledging the different sources of variability in the prediction process.

Minimizing Mean Squared Error

Minimizing mean squared error (MSE) is a common objective in statistical modeling, denoting the average of the squares of the errors. The mean squared error represents the difference between the estimator and what is estimated. In a simple formula, it is \( MSE = E[(Y - \hat{Y})^2] \), where \( Y \) is the true value and \( \hat{Y} \) is the predicted value.
Minimization of MSE is sought in predictive modeling to enhance accuracy; it's a measure of the quality of an estimator—it is always non-negative, and values closer to zero are better. In the context of this exercise, minimizing the MSE equates to finding the predictor \( Y_i \) with the smallest difference from \( X \) on average, which will have the lowest variance according to our calculations. Hence, the predictor \( Y_i \) that has the smallest variance will produce the smallest MSE, indicating it is the most accurate predictor among the available choices.