Question

Suppose \(\boldsymbol{Y}\) is an \(n \times 1\) random vector, \(\boldsymbol{X}\) is an \(n \times p\) matrix of known constants of rank \(p\), and \(\beta\) is a \(p \times 1\) vector of regression coefficients. Let \(\boldsymbol{Y}\) have a \(N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}\right)\) distribution. Discuss the joint pdf of \(\hat{\boldsymbol{\beta}}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{Y}\) and \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\)

Short answer

The joint pdf of the regression coefficient estimator \(\hat{\boldsymbol{\beta}}\) and the estimator for residual sum of squares \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\) is the product of their respective distributions. \(\hat{\boldsymbol{\beta}}\) follows a multivariate normal distribution while the residual sum of squares follows a chi-square distribution with n-p degrees of freedom.

Step-by-step solution

Joint Distribution

The distribution of the vector \(\boldsymbol{Y}\) is given by the following equation: \(\boldsymbol{Y} \sim N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}\right)\). The normal equations of the linear regression model are \(\boldsymbol{X}^{\prime}\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}^{\prime} \boldsymbol{Y}\), which is the basis of finding \(\hat{\boldsymbol{\beta}}\).

Estimator for Regression Coefficients

Solving the normal equations for \(\hat{\boldsymbol{\beta}}\), we find that \(\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1}\boldsymbol{X}^{\prime} \boldsymbol{Y}\). Therefore, the distribution for \(\hat{\boldsymbol{\beta}}\) is as follows: \(\hat{\boldsymbol{\beta}} \sim N\left(\boldsymbol{\beta}, \sigma^{2}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1}\right)\)

Estimator for Residual Sum of Squares

The quantity \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\) represents the residual sum of squares. It can be shown that, with degrees of freedom n-p, it follows a chi-square distribution, represented as: \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2} \sim \chi^{2}_{(n-p)}\)

Joint pdf

We need to identify the joint pdf of \(\hat{\boldsymbol{\beta}}\) and \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\). As these quantities are independent and derived from a normally distributed random vector, the joint pdf would be the product of their individual pdfs, which are normal and chi-square respectively.