Logout Open In App
Open In App
Warning: foreach() argument must be of type array|object, bool given in /var/www/html/web/app/themes/studypress-core-theme/template-parts/header/mobile-offcanvas.php on line 20

Chapter 9: Problem 15

Suppose \(\boldsymbol{Y}\) is an \(n \times 1\) random vector, \(\boldsymbol{X}\) is an \(n \times p\) matrix of known constants of rank \(p\), and \(\beta\) is a \(p \times 1\) vector of regression coefficients. Let \(\boldsymbol{Y}\) have a \(N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}\right)\) distribution. Discuss the joint pdf of \(\hat{\boldsymbol{\beta}}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{Y}\) and \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\)

Short Answer

Expert verified
The joint pdf of the regression coefficient estimator \(\hat{\boldsymbol{\beta}}\) and the estimator for residual sum of squares \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\) is the product of their respective distributions. \(\hat{\boldsymbol{\beta}}\) follows a multivariate normal distribution while the residual sum of squares follows a chi-square distribution with n-p degrees of freedom.

Step by step solution

01

Joint Distribution

The distribution of the vector \(\boldsymbol{Y}\) is given by the following equation: \(\boldsymbol{Y} \sim N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}\right)\). The normal equations of the linear regression model are \(\boldsymbol{X}^{\prime}\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}^{\prime} \boldsymbol{Y}\), which is the basis of finding \(\hat{\boldsymbol{\beta}}\).
02

Estimator for Regression Coefficients

Solving the normal equations for \(\hat{\boldsymbol{\beta}}\), we find that \(\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1}\boldsymbol{X}^{\prime} \boldsymbol{Y}\). Therefore, the distribution for \(\hat{\boldsymbol{\beta}}\) is as follows: \(\hat{\boldsymbol{\beta}} \sim N\left(\boldsymbol{\beta}, \sigma^{2}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1}\right)\)
03

Estimator for Residual Sum of Squares

The quantity \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\) represents the residual sum of squares. It can be shown that, with degrees of freedom n-p, it follows a chi-square distribution, represented as: \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2} \sim \chi^{2}_{(n-p)}\)
04

Joint pdf

We need to identify the joint pdf of \(\hat{\boldsymbol{\beta}}\) and \(\boldsymbol{Y}^{\prime}\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime}\right] \boldsymbol{Y} / \sigma^{2}\). As these quantities are independent and derived from a normally distributed random vector, the joint pdf would be the product of their individual pdfs, which are normal and chi-square respectively.

Unlock Step-by-Step Solutions & Ace Your Exams!

  • Full Textbook Solutions

    Get detailed explanations and key concepts

  • Unlimited Al creation

    Al flashcards, explanations, exams and more...

  • Ads-free access

    To over 500 millions flashcards

  • Money-back guarantee

    We refund you if you fail your exam.

Start your free trial

Over 30 million students worldwide already upgrade their learning with Vaia!

Key Concepts

These are the key concepts you need to understand to accurately answer the question.

Linear Regression
Linear regression is a statistical method used to model the relationship between a dependent variable and one or more independent variables. It is represented by the equation:\[Y = X \beta + \epsilon\]where
  • \(Y\) is the dependent variable vector.
  • \(X\) is the matrix of independent variables (also known as the design matrix).
  • \(\beta\) is the vector of regression coefficients.
  • \(\epsilon\) represents the errors or residuals, assumed to be normally distributed.
The goal of linear regression is to find the best-fitting line through the data. This line minimizes the discrepancy between the observed values and the values predicted by the model. This process is often visualized graphically as finding the line that best fits the points in a scatterplot of the data. Linear regression works under the assumption that these residuals are normally distributed, which simplifies the computation and interpretation of the various statistical inferences related to the model. Linear regression is widely used because it provides a simple yet powerful framework to predict outcomes and establish relationships between variables.
Normal Equations
The normal equations are a set of equations that provide the solution for the coefficients in a linear regression model. They originate from setting the derivative of the sum of squared residuals equal to zero. Mathematically, the normal equations can be expressed as:\[X'X\hat{\beta} = X'Y\]To solve for \(\hat{\beta}\), the regression coefficients, you rearrange the equation:\[\hat{\beta} = (X'X)^{-1}X'Y\]where
  • \(X'X\) is a \(p \times p\) matrix derived from the dot product of the transpose of \(X\) with \(X\).
  • \(X'Y\) is the dot product of the transpose of \(X\) with \(Y\).
  • \((X'X)^{-1}\) is the inverse of \(X'X\), which is crucial to compute \(\hat{\beta}\).
The estimation of \(\hat{\beta}\) through these equations assumes that \(X'X\) is invertible, which typically requires that \(X\) has full column rank, meaning the columns are linearly independent. These normal equations are foundational in the least squares method and ensure that the best possible fit is achieved for the linear regression model under typical assumptions of normality and equal variance of residuals.
Residual Sum of Squares
Residual Sum of Squares (RSS) is a measure of the discrepancy between the data and the estimation model. In the context of regression, residuals are the differences between observed values and the values predicted by the model. Mathematically, RSS can be expressed as:\[RSS = \boldsymbol{Y}'\left[\boldsymbol{I} - \boldsymbol{X}(\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\right] \boldsymbol{Y}\]where
  • \(\boldsymbol{Y}\) is the vector of observed values.
  • \(\boldsymbol{I}\) is the identity matrix of size \(n \times n\).
  • The term \([\boldsymbol{I} - \boldsymbol{X}(\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}']\) projects \(\boldsymbol{Y}\) onto the space orthogonal to the column space of \(\boldsymbol{X}\), effectively isolating the error term.
RSS is used to assess the fit of a linear model; a smaller RSS indicates a better fit. It follows a chi-square distribution with \(n-p\) degrees of freedom, where \(n\) is the number of observations and \(p\) is the number of predictors. The chi-square distribution arises because RSS is essentially a sum of squared normal residuals. RSS is vital in regression analysis as it forms the basis for calculating other important metrics like the coefficient of determination, also known as \(R^2\), and the standard error of the estimate.

One App. One Place for Learning.

All the tools & learning materials you need for study success - in one app.

Get started for free

Most popular questions from this chapter

Often in regression the mean of the random variable \(Y\) is a linear function of \(p\) -values \(x_{1}, x_{2}, \ldots, x_{p}\), say \(\beta_{1} x_{1}+\beta_{2} x_{2}+\cdots+\beta_{p} x_{p}\), where \(\boldsymbol{\beta}^{\prime}=\left(\beta_{1}, \beta_{2}, \ldots, \beta_{p}\right)\) are the regression coefficients. Suppose that \(n\) values, \(\boldsymbol{Y}^{\prime}=\left(Y_{1}, Y_{2}, \ldots, Y_{n}\right)\) are observed for the \(x\) -values in \(\boldsymbol{X}=\left[x_{i j}\right]\), where \(\boldsymbol{X}\) is an \(n \times p\) design matrix and its ith row is associated with \(Y_{i}, i=1,2, \ldots, n .\) Assume that \(Y\) is multivariate normal with mean \(\boldsymbol{X} \boldsymbol{\beta}\) and variance-covariance matrix \(\sigma^{2} \boldsymbol{I}\), where \(\boldsymbol{I}\) is the \(n \times n\) identity matrix. (a) Note that \(Y_{1}, Y_{2}, \ldots, Y_{n}\) are independent. Why? (b) Since \(\boldsymbol{Y}\) should approximately equal its mean \(\boldsymbol{X} \boldsymbol{\beta}\), we estimate \(\boldsymbol{\beta}\) by solving the normal equations \(\boldsymbol{X}^{\prime} \boldsymbol{Y}=\boldsymbol{X}^{\prime} \boldsymbol{X} \boldsymbol{\beta}\) for \(\boldsymbol{\beta}\). Assuming that \(\boldsymbol{X}^{\prime} \boldsymbol{X}\) is non- singular, solve the equations to get \(\hat{\boldsymbol{\beta}}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{Y}\). Show that \(\hat{\boldsymbol{\beta}}\) has a multivariate normal distribution with mean \(\boldsymbol{\beta}\) and variance-covariance matrix $$ \sigma^{2}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} $$ (c) Show that $$ (\boldsymbol{Y}-\boldsymbol{X} \boldsymbol{\beta})^{\prime}(\boldsymbol{Y}-\boldsymbol{X} \boldsymbol{\beta})=(\hat{\boldsymbol{\beta}}-\boldsymbol{\beta})^{\prime}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)(\hat{\boldsymbol{\beta}}-\boldsymbol{\beta})+(\boldsymbol{Y}-\boldsymbol{X} \hat{\boldsymbol{\beta}})^{\prime}(\boldsymbol{Y}-\boldsymbol{X} \hat{\boldsymbol{\beta}}) $$ say \(Q=Q_{1}+Q_{2}\) for convenience. (d) Show that \(Q_{1} / \sigma^{2}\) is \(\chi^{2}(p)\). (e) Show that \(Q_{1}\) and \(Q_{2}\) are independent. (f) Argue that \(Q_{2} / \sigma^{2}\) is \(\chi^{2}(n-p)\). (g) Find \(c\) so that \(c Q_{1} / Q_{2}\) has an \(F\) -distribution. (h) The fact that a value \(d\) can be found so that \(P\left(c Q_{1} / Q_{2} \leq d\right)=1-\alpha\) could be used to find a \(100(1-\alpha)\) percent confidence ellipsoid for \(\beta\). Explain.

If \(A_{1}, A_{2}, \ldots, A_{k}\) are events, prove, by induction, Boole's inequality $$ P\left(A_{1} \cup A_{2} \cup \cdots \cup A_{k}\right) \leq \sum_{1}^{k} P\left(A_{i}\right) $$ Then show that $$ P\left(A_{1}^{c} \cap A_{2}^{c} \cap \cdots \cap A_{k}^{c}\right) \geq 1-\sum_{1}^{b} P\left(A_{i}\right) $$

Let \(X_{1}\) and \(X_{2}\) be two independent random variables. Let \(X_{1}\) and \(Y=\) \(X_{1}+X_{2}\) be \(\chi^{2}\left(r_{1}, \theta_{1}\right)\) and \(\chi^{2}(r, \theta)\), respectively. Here \(r_{1}

Using the notation of this section, assume that the means satisfy the condition that \(\mu=\mu_{1}+(b-1) d=\mu_{2}-d=\mu_{3}-d=\cdots=\mu_{b}-d .\) That is, the last \(b-1\) means are equal but differ from the first mean \(\mu_{1}\), provided that \(d \neq 0\). Let independent random samples of size \(a\) be taken from the \(b\) normal distributions with common unknown variance \(\sigma^{2}\). (a) Show that the maximum likelihood estimators of \(\mu\) and \(d\) are \(\hat{\mu}=\bar{X} . .\) and $$ \hat{d}=\frac{\sum_{j=2}^{b} \bar{X}_{. j} /(b-1)-\bar{X}_{.1}}{b} $$ (b) Using Exercise \(9.1 .3\), find \(Q_{6}\) and \(Q_{7}=c \hat{d}^{2}\) so that, when \(d=0, Q_{7} / \sigma^{2}\) is \(\chi^{2}(1)\) and $$ \sum_{i=1}^{a} \sum_{j=1}^{b}\left(X_{i j}-\bar{X}_{n}\right)^{2}=Q_{3}+Q_{6}+Q_{7} $$ (c) Argue that the three terms in the right-hand member of Part (b), once divided by \(\sigma^{2}\), are independent random variables with chi-square distributions, provided that \(d=0\). (d) The ratio \(Q_{7} /\left(Q_{3}+Q_{6}\right)\) times what constant has an \(F\) -distribution, provided that \(d=0\) ? Note that this \(F\) is really the square of the two-sample \(T\) used to test the equality of the mean of the first distribution and the common mean of the other distributions, in which the last \(b-1\) samples are combined into one.

Here \(Q_{1}\) and \(Q_{2}\) are quadratic forms in observations of a random sample from \(N(0,1) .\) If \(Q_{1}\) and \(Q_{2}\) are independent and if \(Q_{1}+Q_{2}\) has a chi-square distribution, prove that \(Q_{1}\) and \(Q_{2}\) are chi-square variables.
See all solutions

Recommended explanations on Math Textbooks

Theoretical and Mathematical Physics

Read Explanation

Mechanics Maths

Read Explanation

Statistics

Read Explanation

Geometry

Read Explanation

Logic and Functions

Read Explanation

Discrete Mathematics

Read Explanation
View all explanations

What do you think about this solution?

We value your feedback to improve our textbook solutions.

Study anywhere. Anytime. Across all devices.

Sign-up for free