Question

Suppose \(\boldsymbol{Y}\) is an \(n \times 1\) random vector, \(\boldsymbol{X}\) is an \(n \times p\) matrix of known constants of rank \(p\), and \(\beta\) is a \(p \times 1\) vector of regression coefficients. Let \(\boldsymbol{Y}\) have a \(N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}\right)\) distribution. Discuss the joint pdf of \(\hat{\boldsymbol{\beta}}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{Y}\) and \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\)

Short answer

The joint pdf of the regression coefficient estimator \(\hat{\boldsymbol{\beta}}\) and the estimator for residual sum of squares \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\) is the product of their respective distributions. \(\hat{\boldsymbol{\beta}}\) follows a multivariate normal distribution while the residual sum of squares follows a chi-square distribution with n-p degrees of freedom.

Step-by-step solution

Joint Distribution

The distribution of the vector \(\boldsymbol{Y}\) is given by the following equation: \(\boldsymbol{Y} \sim N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}\right)\). The normal equations of the linear regression model are \(\boldsymbol{X}^{\prime}\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}^{\prime} \boldsymbol{Y}\), which is the basis of finding \(\hat{\boldsymbol{\beta}}\).

Estimator for Regression Coefficients

Solving the normal equations for \(\hat{\boldsymbol{\beta}}\), we find that \(\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1}\boldsymbol{X}^{\prime} \boldsymbol{Y}\). Therefore, the distribution for \(\hat{\boldsymbol{\beta}}\) is as follows: \(\hat{\boldsymbol{\beta}} \sim N\left(\boldsymbol{\beta}, \sigma^{2}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1}\right)\)

Estimator for Residual Sum of Squares

The quantity \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\) represents the residual sum of squares. It can be shown that, with degrees of freedom n-p, it follows a chi-square distribution, represented as: \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2} \sim \chi^{2}_{(n-p)}\)

Joint pdf

We need to identify the joint pdf of \(\hat{\boldsymbol{\beta}}\) and \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\). As these quantities are independent and derived from a normally distributed random vector, the joint pdf would be the product of their individual pdfs, which are normal and chi-square respectively.

Key concepts

Linear Regression

Linear regression is a statistical method used to model the relationship between a dependent variable and one or more independent variables. It is represented by the equation:\[Y = X \beta + \epsilon\]where

• \(Y\) is the dependent variable vector.
• \(X\) is the matrix of independent variables (also known as the design matrix).
• \(\beta\) is the vector of regression coefficients.
• \(\epsilon\) represents the errors or residuals, assumed to be normally distributed.

The goal of linear regression is to find the best-fitting line through the data. This line minimizes the discrepancy between the observed values and the values predicted by the model. This process is often visualized graphically as finding the line that best fits the points in a scatterplot of the data. Linear regression works under the assumption that these residuals are normally distributed, which simplifies the computation and interpretation of the various statistical inferences related to the model. Linear regression is widely used because it provides a simple yet powerful framework to predict outcomes and establish relationships between variables.

Normal Equations

The normal equations are a set of equations that provide the solution for the coefficients in a linear regression model. They originate from setting the derivative of the sum of squared residuals equal to zero. Mathematically, the normal equations can be expressed as:\[X'X\hat{\beta} = X'Y\]To solve for \(\hat{\beta}\), the regression coefficients, you rearrange the equation:\[\hat{\beta} = (X'X)^{-1}X'Y\]where

• \(X'X\) is a \(p \times p\) matrix derived from the dot product of the transpose of \(X\) with \(X\).
• \(X'Y\) is the dot product of the transpose of \(X\) with \(Y\).
• \((X'X)^{-1}\) is the inverse of \(X'X\), which is crucial to compute \(\hat{\beta}\).

The estimation of \(\hat{\beta}\) through these equations assumes that \(X'X\) is invertible, which typically requires that \(X\) has full column rank, meaning the columns are linearly independent. These normal equations are foundational in the least squares method and ensure that the best possible fit is achieved for the linear regression model under typical assumptions of normality and equal variance of residuals.

Residual Sum of Squares

Residual Sum of Squares (RSS) is a measure of the discrepancy between the data and the estimation model. In the context of regression, residuals are the differences between observed values and the values predicted by the model. Mathematically, RSS can be expressed as:\[RSS = \boldsymbol{Y}'\left[\boldsymbol{I} - \boldsymbol{X}(\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\right] \boldsymbol{Y}\]where

• \(\boldsymbol{Y}\) is the vector of observed values.
• \(\boldsymbol{I}\) is the identity matrix of size \(n \times n\).
• The term \([\boldsymbol{I} - \boldsymbol{X}(\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}']\) projects \(\boldsymbol{Y}\) onto the space orthogonal to the column space of \(\boldsymbol{X}\), effectively isolating the error term.

RSS is used to assess the fit of a linear model; a smaller RSS indicates a better fit. It follows a chi-square distribution with \(n-p\) degrees of freedom, where \(n\) is the number of observations and \(p\) is the number of predictors. The chi-square distribution arises because RSS is essentially a sum of squared normal residuals. RSS is vital in regression analysis as it forms the basis for calculating other important metrics like the coefficient of determination, also known as \(R^2\), and the standard error of the estimate.