Question

(a) Consider a normal linear model \(y=X \beta+\varepsilon\) where \(\operatorname{var}(\varepsilon)=\sigma^{2} W^{-1}\), and \(W\) is a known positive definite symmetric matrix. Show that a inverse square root matrix \(W^{1 / 2}\) exists, and re-express the least squares problem in terms of \(y_{11}=W^{1 / 2} y, X_{1}=W^{1 / 2} X\), and \(\varepsilon_{1}=W^{1 / 2} \varepsilon .\) Show that \(\operatorname{var}\left(\varepsilon_{1}\right)=\sigma^{2} I_{n} .\) Hence find the least squares estimates, hat matrix, and residual sum of squares for the weighted regression in terms of \(y, X\), and \(W\), and give the distributions of the least squares estimates of \(\beta\) and the residual sum of squares. (b) Suppose that \(W\) depends on an unknown scalar parameter, \(\rho\). Find the profile log likelihood for \(\rho, \ell_{\mathrm{p}}(\rho)=\max _{\beta, \sigma^{2}} \ell\left(\beta, \sigma^{2}, \rho\right)\), and outline how to use a least squares package to give a confidence interval for \(\rho\).

Short answer

The least squares estimator is \( \hat{\beta} = (X^T W X)^{-1} X^T W y \). The RSS is \( (y - X \hat{\beta})^T W (y - X \hat{\beta}) \). For \( \rho \), use the profile log-likelihood to estimate and construct a confidence interval.

Step-by-step solution

Verify the Existence of Inverse Square Root Matrix

We start with the matrix \( W \) which is known to be a positive definite symmetric matrix. For any positive definite matrix, there exists a unique inverse square root matrix, \( W^{-1/2} \). This can be achieved by the spectral decomposition of \( W = Q \Lambda Q^T \), where \( Q \) is an orthogonal matrix and \( \Lambda \) is a diagonal matrix with positive eigenvalues. Hence, \( W^{-1/2} = Q \Lambda^{-1/2} Q^T \) exists.

Re-expressing the Model

The normal linear model is given by \( y = X \beta + \varepsilon \). By transforming the variables using \( W^{1/2} \), we define the new variables as \( y_{1} = W^{1/2} y \), \( X_{1} = W^{1/2} X \), and \( \varepsilon_{1} = W^{1/2} \varepsilon \). Thus, the transformed model becomes \( y_{1} = X_{1} \beta + \varepsilon_{1} \).

Simplifying Variance of Transformed Errors

The variance of the error term \( \varepsilon \) in the original model is \( \sigma^2 W^{-1} \). Transforming \( \varepsilon \) gives \( \varepsilon_{1} = W^{1/2} \varepsilon \). The variance becomes \( \operatorname{var}(\varepsilon_{1}) = W^{1/2} \operatorname{var}(\varepsilon) W^{1/2} = W^{1/2} (\sigma^2 W^{-1}) W^{1/2} = \sigma^2 I_n \) since multiplying by \( W^{1/2} \) and its inverse, \( W^{-1/2} \), results in the identity matrix \( I_n \).

Solving the Least Squares Problem

In the transformed model \( y_{1} = X_{1} \beta + \varepsilon_{1} \), the least squares estimator \( \hat{\beta} \) is given by \( \hat{\beta} = (X_{1}^T X_{1})^{-1} X_{1}^T y_{1} \). Substituting back the transformations, we have \( \hat{\beta} = (X^T W X)^{-1} X^T W y \).

Determine Hat Matrix and Residual Sum of Squares

The hat matrix \( H \) in the transformed model is \( H = X_{1} (X_{1}^T X_{1})^{-1} X_{1}^T = X (X^T W X)^{-1} X^T W \). The residual sum of squares (RSS) is calculated as \( \text{RSS} = (y_{1} - X_{1} \hat{\beta})^T (y_{1} - X_{1} \hat{\beta}) = (y - X \hat{\beta})^T W (y - X \hat{\beta}) \).

Distribution of Estimates

Under the assumptions of the normal linear model, \( \hat{\beta} \) follows a multivariate normal distribution \( N(\beta, \sigma^2 (X^T W X)^{-1}) \). The scaled residual sum of squares follows a chi-squared distribution: \( \frac{\text{RSS}}{\sigma^2} \sim \chi^2(n-p) \), where \( n \) is the number of observations and \( p \) is the number of parameters in \( \beta \).

Profile Log-Likelihood for Parameter \( \rho \)

To find the profile log likelihood for \( \rho \), we consider the likelihood function of the linear model and replace \( W \) with a function of \( \rho \), say \( W(\rho) \). The profile log likelihood \( \ell_p(\rho) \) is obtained by maximizing the log likelihood \( \ell(\beta, \sigma^2, \rho) \) with respect to \( \beta \) and \( \sigma^2 \), keeping \( \rho \) fixed. The expression is \( -\frac{1}{2} \left(n \log(2\pi \sigma^2) + \log|W| + \frac{(y - X \hat{\beta})^T W (y - X \hat{\beta})}{\sigma^2} \right) \).

Confidence Interval for \( \rho \) Using Least Squares

To find a confidence interval for \( \rho \), we can use a standard least squares package to fit the model for various values of \( \rho \). The \( \rho \) value that maximizes the profile log likelihood corresponds to the point estimate. A confidence interval can be obtained using a likelihood ratio test or by examining values of the log-likelihood function close to its maximum to achieve a specified significance level.

Key concepts

Statistical Inference

Statistical Inference in the context of a normal linear model involves drawing conclusions about population parameters based on a sample. For example, in our model, the parameters of interest are often the coefficient vector \(\beta\) and error variance \(\sigma^{2}\).

In practical terms, statistical inference allows us to make predictions, estimate parameters, and test hypotheses. By analyzing sample data, we aim to infer the population's characteristics. Our model helps us derive estimations about \(\beta\) and \(\sigma^{2}\) using techniques like the least squares estimation and analyzing the transformed model.

Key components of statistical inference in our linear model include:

• Parameter Estimation: Estimating the unknown \(\beta\) and the variance \(\sigma^{2}\).
• Prediction: Predicting new responses for given input values of \(X\).
• Hypothesis Testing: Testing hypotheses about the relationships modeled by \(X\) and \(y\).

This model validation through sample data analysis is the essence of statistical inference, providing a scientific basis for conclusions drawn from the linear model.

Least Squares Estimation

Least Squares Estimation is a core technique for finding the best-fitting line through points in a data set by minimizing the sum of the squares of the residuals. The residuals are differences between observed and predicted values.

In the context of the given normal linear model, we apply least squares estimation to both the original and the transformed model. For the original model \(y = X \beta + \varepsilon\), the least squares estimator \(\hat{\beta}\) is calculated as \((X^T W X)^{-1} X^T W y\).

Through the transformations involving \(W^{1/2}\), we redefine the variables as \(y_{1}\), \(X_{1}\), and \(\varepsilon_{1}\). The least squares estimation process then incorporates these transformed variables to maintain the optimization criterion. This method ensures that the parameter estimates minimize the discrepancies between observed data and model predictions, enhancing the model accuracy.

• Formulation: Express parameters in terms of observable data using transformation.
• Residual Minimization: Adjust estimates until the sum of squared differences between observed and predicted values is minimal.

Understanding this optimization technique is crucial for accurately estimating coefficients of the linear model.

Weighted Regression

Weighted Regression is an enhancement of ordinary least squares regression where observations are given different weights. This method is particularly useful when the variance of the errors is not constant.

In our model, the weighting comes from the matrix \(W\), a positive definite symmetric matrix. This matrix adjusts the influence of each data point on the parameter estimates, essentially treating some observations as more informative than others.
By transforming the original model using the inverse square root of \(W\), we achieve homoscedasticity in the variance of the errors, simplifying them to \(\sigma^{2} I_{n}\). This transformation allows for the same least squares approach to be applied while accounting for variance structures within the data.

Advantages of using weighted regression include:

• Improved Estimation: Better parameter estimation by correcting variance differences.
• Accuracy: More accurate predictions in the presence of heteroscedasticity.
• Flexibility: Ability to transform models and improve fit with weighted adjustments.

Weighted regression ensures that each data point contributes appropriately to parameter estimates, optimizing model precision.

Matrix Decomposition

Matrix Decomposition is the process of breaking a matrix into its constituent parts, which simplifies complex matrix operations. Among the various types, the spectral decomposition used here is essential for handling positive definite matrices like \(W\).

For the spectral decomposition, we express \(W\) as \(Q \Lambda Q^T\), where \(Q\) is an orthogonal matrix and \(\Lambda\) is a diagonal matrix of eigenvalues. This decomposition helps verify the existence of \(W^{-1/2}\), crucial for re-expressing the linear model.

The decomposition serves three main purposes in our context:

• Finding Inverses: Simplifies the computation of matrix inverses especially for complex matrices.
• Diagonalization: Converts complex operations into easier ones using diagonal matrices.
• Model Transformations: Facilitates model transformations, enabling the use of weighted regression by easing computation through simplified forms.

By mastering this technique, handling weight matrices and transforming models become more manageable, leading to more efficient data analysis.