Question

Suppose that random variables \(Y_{g j}, j=1, \ldots, n_{g}, g=1, \ldots, G\), are independent and that they satisfy the normal linear model \(Y_{g j}=x_{g}^{\mathrm{T}} \beta+\varepsilon_{g j}\). Write down the covariate matrix for this model, and show that the least squares estimates can be written as \(\left(X_{1}^{\mathrm{T}} W X_{1}\right)^{-1} X_{1}^{\mathrm{T}} W Z\), where \(W=\operatorname{diag}\left\\{n_{1}, \ldots, n_{G}\right\\}\), and the \(g\) th element of \(Z\) is \(n_{g}^{-1} \sum_{j} Y_{g j} .\) Hence show that weighted least squares based on \(Z\) and unweighted least squares based on \(Y\) give the same parameter estimates and confidence intervals, when \(\sigma^{2}\) is known. Why do they differ if \(\sigma^{2}\) is unknown, unless \(n_{g} \equiv 1 ?\) Discuss how the residuals for the two setups differ, and say which is preferable for modelchecking.

Short answer

Weighted and unweighted least squares give the same estimates when \(\sigma^2\) is known. For model-checking, the unweighted residuals are preferable due to their finer granularity.

Step-by-step solution

Define the Covariate Matrix

In the normal linear model given by \(Y_{g j} = x_{g}^{\mathrm{T}} \beta + \varepsilon_{g j}\), the covariate matrix \(X\) is constructed from the vectors \(x_g\) which contain the covariates for each group. If \(x_g\) is a column vector for group \(g\), then the full matrix \(X\) will be a \(n \times p\) matrix, where each block corresponds to the expanded covariate for group \(g\). For simplicity, assume all \(n_g\) are arranged in order in \(X\).

Determine Weighted Matrix W and Vector Z

The matrix \(W\) is the diagonal matrix \(\operatorname{diag}\{n_1, \ldots, n_G\}\), which weighs each group according to its size. The vector \(Z\) has entries \(Z_g = n_g^{-1} \sum_j Y_{g j}\), which is the average of the observations in each group. This transformation aggregates observations into single outputs per group.

Express Least Squares with X, W, Z

The least squares estimates are given by minimizing the residual sum of squares. For weighted least squares, this estimate can be found as \(\left(X_{1}^{\mathrm{T}} W X_{1}\right)^{-1} X_{1}^{\mathrm{T}} W Z\). Here, \(X_1\) is the adjusted covariate matrix for alignment with aggregated data in \(Z\). Substitute the given form of \(W\) and \(Z\) to show the relationship.

Comparison for Known \(\sigma^2\)

When \(\sigma^2\) is known, both the weighted least squares using \(Z\) and the unweighted least squares using unaggregated \(Y\) produce identical parameter estimates and identical confidence intervals, due to perfect balancing through known variance. Aggregation and individual weighting align calculations.

Impact for Unknown \(\sigma^2\) and Residual Analysis

If \(\sigma^2\) is unknown, then weighting introduces variability in estimated variance of residuals, making the models differ. If \(n_g \equiv 1\) for all groups, no aggregation occurs, leading to equivalent models. The weighted design can obscure residual diagnostics due to averaging effects; unweighted residuals in \(Y\) provide detailed diagnostics and better outlier detection.

Key concepts

Understanding the Covariate Matrix

In the context of linear regression, the covariate matrix, often denoted as \( X \), plays a pivotal role. It's essentially a structured way of organizing all the input data or independent variables for regression analysis. The covariate matrix is essential because it serves as the basis for estimating parameters in the linear model.
Consider a model where we have several groups, each with its own set of observations and associated covariates. In the given exercise, each group \( g \) has a vector \( x_g \) containing relevant covariates. By stacking these vectors for each group into a larger matrix, we form the full covariate matrix \( X \).
This matrix is "block-structured," meaning entries from each group are organized in sequence. If you imagine these blocks as building blocks, each block corresponds to a group with its associated covariate data included in the analysis. The matrix \( X \) is a crucial component because it directly influences how we estimate the relationships between variables in our model.

Diving into Least Squares Estimation

Least squares estimation is a fundamental technique in statistical modeling, particularly within linear regression. The core idea behind least squares is to find the parameter estimates that minimize the sum of the squared differences between the observed values and the values predicted by the model.
The method gets its name from the way residuals (the differences between observed and predicted values) are squared and then summed. By minimizing this sum, we obtain what are known as least squares estimates. These estimates are expressed mathematically as \( \hat{\beta} = (X^{\mathrm{T}} X)^{-1} X^{\mathrm{T}} Y \), which derives from setting up the normal equations for the linear model and solving for the parameter vector \( \beta \).
This estimation procedure is incredibly useful because it provides a straightforward way to find best-fit parameters for the model, ensuring that predicted outcomes are as close as possible to the actual data collected.

Exploring Weighted Least Squares Estimation

Weighted least squares (WLS) is an extension of the least squares estimation technique, where weights are assigned to each observation. This approach becomes particularly handy when different observations have differing levels of reliability or variance.
In weighted least squares, the principle remains the same—minimizing the sum of squared residuals—but with an important twist: each residual is weighted according to a specific scheme. For the exercise at hand, the weights are provided by the matrix \( W = \operatorname{diag}\{n_1, \ldots, n_G\} \).
These weights help account for some inherent variability or group size differences among observations. This weighted approach provides a more nuanced parameter estimation compared to ordinary least squares, especially when observations are not equally reliable or when there's variance heterogeneity.
When \( \sigma^2 \), the variance, is known, weighted and unweighted models can yield identical results, distributed accurately by known variances. However, if \( \sigma^2 \) is unknown, the model can yield different outcomes due to the introduced variability in residuals. Here, unweighted analysis often leads to more reliable residual diagnostics, offering clearer insights into model fit and potential outliers.