Question

Often in regression the mean of the random variable \(Y\) is a linear function of \(p\) -values \(x_{1}, x_{2}, \ldots, x_{p}\), say \(\beta_{1} x_{1}+\beta_{2} x_{2}+\cdots+\beta_{p} x_{p}\), where \(\beta^{\prime}=\left(\beta_{1}, \beta_{2}, \ldots, \beta_{p}\right)\) are the regression coefficients. Suppose that \(n\) values, \(\boldsymbol{Y}^{\prime}=\left(Y_{1}, Y_{2}, \ldots, Y_{n}\right)\), are observed for the \(x\) -values in \(\boldsymbol{X}=\left[x_{i j}\right]\), where \(\boldsymbol{X}\) is an \(n \times p\) design matrix and its ith row is associated with \(Y_{i}, i=1,2, \ldots, n\). Assume that \(\boldsymbol{Y}\) is multivariate normal with mean \(\boldsymbol{X} \boldsymbol{\beta}\) and variance-covariance matrix \(\sigma^{2} \boldsymbol{I}\), where \(\boldsymbol{I}\) is the \(n \times n\) identity matrix. (a) Note that \(Y_{1}, Y_{2}, \ldots, Y_{n}\) are independent. Why? (b) Since \(\boldsymbol{Y}\) should approximately equal its mean \(\boldsymbol{X} \boldsymbol{\beta}\), we estimate \(\boldsymbol{\beta}\) by solving the normal equations \(\boldsymbol{X}^{\prime} \boldsymbol{Y}=\boldsymbol{X}^{\prime} \boldsymbol{X} \boldsymbol{\beta}\) for \(\boldsymbol{\beta}\). Assuming that \(\boldsymbol{X}^{\prime} \boldsymbol{X}\) is non- singular, solve the equations to get \(\hat{\beta}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{Y}\). Show that \(\hat{\boldsymbol{\beta}}\) has a multivariate normal distribution with mean \(\boldsymbol{\beta}\) and variance- covariance matrix $$ \sigma^{2}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} $$ (c) Show that $$ (\boldsymbol{Y}-\boldsymbol{X} \boldsymbol{\beta})^{\prime}(\boldsymbol{Y}-\boldsymbol{X} \boldsymbol{\beta})=(\hat{\boldsymbol{\beta}}-\boldsymbol{\beta})^{\prime}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)(\hat{\boldsymbol{\beta}}-\boldsymbol{\beta})+(\boldsymbol{Y}-\boldsymbol{X} \hat{\boldsymbol{\beta}})^{\prime}(\boldsymbol{Y}-\boldsymbol{X} \hat{\boldsymbol{\beta}}) $$ For the remainder of the exercise, let \(Q\) denote the quadratic form on the left side of this expression and \(Q_{1}\) and \(Q_{2}\) denote the respective quadratic forms on the right side. Hence, \(Q=Q_{1}+Q_{2}\). (d) Show that \(Q_{1} / \sigma^{2}\) is \(\chi^{2}(p)\). (e) Show that \(Q_{1}\) and \(Q_{2}\) are independent. (f) Argue that \(Q_{2} / \sigma^{2}\) is \(\chi^{2}(n-p)\). (g) Find \(c\) so that \(c Q_{1} / Q_{2}\) has an \(F\) -distribution. (h) The fact that a value \(d\) can be found so that \(P\left(c Q_{1} / Q_{2} \leq d\right)=1-\alpha\) could be used to find a \(100(1-\alpha) \%\) confidence ellipsoid for \(\boldsymbol{\beta} .\) Explain.

Short answer

The values \(Y_{1}, Y_{2}, ..., Y_{n}\) are independent because the covariance matrix is a scalar multiple of the identity matrix. The estimator \(\hat{\beta}\) is derived as \((X^TX)^{-1}X^TY\) and follows a multivariate normal distribution. The given quadratic forms equation is proven through algebraic manipulations. \(Q_{1}\) and \(Q_{2}\), are independently distributed as \(\chi^2\) with \(p\) and \(n - p\) degrees of freedom respectively. A constant \(c\) is found such that \(cQ_1/Q_2\) has an F-distribution. Finally, a boundary value \(d\) is used to construct a 100(1 - \alpha)% confidence ellipsoid for \(\beta\).

Step-by-step solution

Reasoning about the independence of \(Y_{1}, Y_{2}, \ldots, Y_{n}\)

Examine the given scenario. It is stated that the vector \(Y\) is multivariate normal with mean \(X\beta\) and variance-covariance matrix \(\sigma^{2}I\), where \(I\) is the identity matrix. The diagonal elements of the covariance matrix represent the variances of the respective variables, and the off-diagonal elements represent the covariances. A covariance of zero between two variables indicates that they are statistically independent. Hence, since all off-diagonal elements of the matrix \(\sigma^{2}I\) are zero, all \(Y_{1}, Y_{2}, \ldots, Y_{n}\) are independent.

Solving for \(\hat{\beta}\) and its distribution

As given, solve the normal equations by setting \(X^TY = X^TX\beta\). From this, it can be derived that \(\hat{\beta} = (X^TX)^{-1}X^TY\), which is the ordinary least squares estimator for \(\beta\). Since \(\hat{\beta}\) is a linear transformation of the multivariate normally distributed variable \(Y\), it also follows a multivariate normal distribution with mean \(\beta\) and covariance matrix \(\sigma^{2}(X^TX)^{-1}\) by the properties of the multivariate normal distribution and linear transformations of it.

Proving the equality of quadratic forms

Define and prove the equality of quadratic forms through algebraic manipulations and the properties of transposes. Consider statements and properties such as \((A + B)^T = A^T + B^T\), \((AB)^T = B^TA^T\), \((A + B)^T(A + B) = A^TA + A^TB + B^TA + B^TB\), and \(X^TX\hat{\beta} = X^TY\). Combine these properties and statements to prove the provided equation.

Distributing chi-squared distributions to \(Q_{1}\) and \(Q_{2}\)

Consider that \(Q_1\) and \(Q_2\) are respectively chi-squared distributed with \(p\) and \(n - p\) degrees of freedom, because they are the sum of squared independent and normally distributed variables divided by their variances.

Discussing the independence of \(Q_{1}\) and \(Q_{2}\)

State that \(Q_1\) and \(Q_2\) are independent, as they are functions of disjoint sets of independent variables.

Applying the F-distribution

Recall the definition of the F-distribution which is the ratio of two independent chi-squared variables divided by their respective degrees of freedom. From this definition, find a constant \(c\) such that \(cQ_1/Q_2\) follows an F-distribution.

Interpreting confidence ellipsoid

Explain how determining a boundary value \(d\) such that the probability \(P(cQ_1/Q_2 \leq d) = 1 - \alpha\) can be used to define a 100(1 - \alpha)% confidence ellipsoid for \(\beta\). This probability indicates that, within the confidence ellipsoid, the true value of \(\beta\) can be found with a confidence level of 100(1 - \alpha)%, thus providing a range of plausible values for the regression parameters.