Question

Suppose that the random variables \(Y_{1}, \ldots, Y_{n}\) are such that $$ \mathrm{E}\left(Y_{j}\right)=\mu, \quad \operatorname{var}\left(Y_{j}\right)=\sigma_{j}^{2}, \quad \operatorname{cov}\left(Y_{j}, Y_{k}\right)=0, \quad j \neq k $$ where \(\mu\) is unknown and the \(\sigma_{j}^{2}\) are known. Show that the linear combination of the \(Y_{j}\) 's giving an unbiased estimator of \(\mu\) with minimum variance is $$ \sum_{j=1}^{n} \sigma_{j}^{-2} Y_{j} / \sum_{j=1}^{n} \sigma_{j}^{-2} $$ Suppose now that \(Y_{j}\) is normally distributed with mean \(\beta x_{j}\) and unit variance, and that the \(Y_{j}\) are independent, with \(\beta\) an unknown parameter and the \(x_{j}\) known constants. Which of the estimators $$ T_{1}=n^{-1} \sum_{j=1}^{n} Y_{j} / x_{j}, \quad T_{2}=\sum_{j=1}^{n} Y_{j} x_{j} / \sum_{j=1}^{n} x_{j}^{2} $$ is preferable and why?

Short answer

The minimum variance unbiased estimator is \( \frac{\sum_{j=1}^{n} \sigma_j^{-2} Y_{j}}{\sum_{j=1}^{n} \sigma_j^{-2}} \), and \( T_2 \) is preferable due to lower variance.

Step-by-step solution

Identify the Weights for Minimum Variance

We need to find the weights that will minimize the variance of the estimator, while maintaining it as an unbiased estimator. For an estimator \( \sum_{j=1}^{n} w_j Y_{j} \), the expected value is \( \mu \) when \( \sum_{j=1}^{n} w_j = 1 \).

Determine Variance of the Estimator

The variance of \( \sum_{j=1}^{n} w_j Y_{j} \) is given by \( \sum_{j=1}^{n} w_j^2 \sigma_j^2 \) due to the independence and zero covariance. Our goal is to minimize this variance.

Use Lagrange Multipliers for Optimization

To minimize \( \sum_{j=1}^{n} w_j^2 \sigma_j^2 \) subject to \( \sum_{j=1}^{n} w_j = 1 \), use the Lagrange multiplier approach. Define the Lagrangian as \( \mathcal{L} = \sum_{j=1}^{n} w_j^2 \sigma_j^2 + \lambda (1 - \sum_{j=1}^{n} w_j) \).

Solve the Lagrange Multiplier Equations

Differentiating \( \mathcal{L} \) with respect to \( w_j \) and \( \lambda \), we get \( 2w_j \sigma_j^2 - \lambda = 0 \) and \( \sum_{j=1}^{n} w_j = 1 \). From the first equation, \( w_j = \frac{\lambda}{2\sigma_j^2} \).

Find \( \lambda \) and the Optimal Weights

Substitute \( w_j = \frac{\lambda}{2\sigma_j^2} \) into \( \sum_{j=1}^{n} w_j = 1 \), we have \( \sum_{j=1}^{n} \frac{\lambda}{2\sigma_j^2} = 1 \). Therefore, \( \lambda = \left( \sum_{j=1}^{n} \sigma_j^{-2} \right)^{-1} \). Hence, \( w_j = \frac{\sigma_j^{-2}}{\sum_{k=1}^{n} \sigma_k^{-2}} \).

Plug Weights into the Estimator

The minimum variance unbiased estimator thus becomes \( \sum_{j=1}^{n} \frac{\sigma_j^{-2}}{\sum_{k=1}^{n} \sigma_k^{-2}} Y_j = \frac{\sum_{j=1}^{n} \sigma_j^{-2} Y_j}{\sum_{j=1}^{n} \sigma_j^{-2}} \).

Compare the Estimators for \( \beta \)

For the normal distribution part, we compare the variances of the estimators \( T_1 \) and \( T_2 \). Given \( Y_j \sim N(\beta x_j, 1) \), \( \mathrm{Var}(T_1) \) is \( \sum_{j} \frac{1}{(x_j)^2} \) while \( \mathrm{Var}(T_2) \) is \( \frac{1}{\sum_{j} (x_j)^2} \).

Conclusion on the Preferable Estimator

Since \( T_2 \) has variance \( \frac{1}{\sum_{j=1}^{n} (x_j)^2} \), which is typically smaller than \( \sum_{j=1}^{n} \frac{1}{(x_j)^2} \) when \( x_j \) are not all equal, \( T_2 \) tends to be preferable due to lower variance.

Key concepts

Minimum Variance Unbiased Estimator

Understanding the concept of a Minimum Variance Unbiased Estimator (MVUE) is crucial in estimation theory. The MVUE is an estimator that satisfies two key properties: it has the smallest variance among all unbiased estimators for a parameter, and it remains unbiased, meaning its expected value is equal to the true parameter value.

To find a MVUE for a given situation, one typically sets up an estimator as a linear combination of data points, and then seeks the weights that minimize estimator variance. This involves ensuring the sum of the weights equals one. Subsequently, by optimally choosing weights, the variance of the estimator is reduced.

For instance, given random variables with known variances, we can construct an unbiased estimator by determining the right weights, calculated as inversely proportional to the known variances of these variables. This guarantees minimized overall estimation variance, achieving what is known as the MVUE.

Lagrange Multipliers

Lagrange multipliers are a potent mathematical tool used to find the extrema of functions subject to constraints. When seeking a Minimum Variance Unbiased Estimator, we often employ this method.

In our exercise, to minimize the variance of the estimator while keeping it unbiased, we apply Lagrange multipliers. We first define the Lagrangian function. This function combines the estimator's variance and the constraint that the sum of the weights equals one. It looks like this: \[ \mathcal{L} = \sum_{j=1}^{n} w_j^2 \sigma_j^2 + \lambda (1 - \sum_{j=1}^{n} w_j) \]

By differentiating this function with respect to both the weights and the Lagrange multiplier, we derive equations that help us determine the optimal weights. This is a cornerstone in optimizing under constraints, revealing the weights with which our estimator attains optimal precision.

Normal Distribution

The normal distribution is a fundamental concept in statistics, characterized by its bell-shaped curve. This distribution is described by two parameters: the mean and the variance.

In the context of our exercise, we consider normally distributed random variables where each variable follows \(Y_j \sim N(\beta x_j, 1)\). Here, the mean of \(Y_j\) is \(\beta x_j\), and these means form a linear relationship influenced by a common parameter \(\beta\). This common parameter is what we aim to estimate based on available samples.

Normal distributions are pivotal in estimation theories because many estimators and tests are based on normally distributed assumptions. This facilitates simplifications in variance computation and comparison, enabling more straightforward derivations of optimal estimations.

Variance Comparison

When comparing estimators, their variances often dictate which estimator is preferable. Lower variance implies higher precision and reliability.

Within our exercise, estimators \(T_1\) and \(T_2\) exhibit different structural traits leading to distinct variances. Specifically, \(T_1\) and \(T_2\) were compared for their abilities to estimate \(\beta\), the common parameter. The variances were calculated as \(\mathrm{Var}(T_1) = \sum_{j} \frac{1}{(x_j)^2}\) and \(\mathrm{Var}(T_2) = \frac{1}{\sum_{j} (x_j)^2}\).

In general, \(T_2\) emerges superior due to its typically lower variance, indicating a more precise estimator. As the sum of squares in the denominator for \(T_2\) generally outweighs individual reciprocals, this relationship results in a smaller variance, affirming \(T_2\) as the preferred estimator unless all weights are equal, where variance comparisons might differ.