Question

Exercises 19 and 20 involve a design matrix \(X\) with two or more columns and a least-squares solution \(\hat \beta \) of \({\bf{y}} = X\beta \). Consider the following numbers.

(i) \({\left\| {X\hat \beta } \right\|^2}\)—the sum of the squares of the “regression term.” Denote this number by \(SS\left( R \right)\).

(ii) \({\left\| {{\bf{y}} - X\hat \beta } \right\|^2}\)—the sum of the squares for error term. Denote this number by \(SS\left( E \right)\).

(iii) \({\left\| {\bf{y}} \right\|^2}\)—the “total” sum of the squares of the -values. Denote this number by \(SS\left( T \right)\).

Every statistics text that discusses regression and the linear model \(y = X\beta + \in \) introduces these numbers, though terminology and notation vary somewhat. To simplify matters, assume that the mean of the -values is zero. In this case, \(SS\left( T \right)\) is proportional to what is called the variance of the set of \(y\)-values.

20. Show that \({\left\| {X\hat \beta } \right\|^2} = {\hat \beta ^T}{X^T}{\bf{y}}\). (Hint: Rewrite the left side and use the fact that \(\hat \beta \) satisfies the normal equations.) This formula for is used in statistics. From this and from Exercise 19, obtain the standard formula for \(SS\left( E \right)\):

\(SS\left( E \right) = {y^T}y - \hat \beta {X^T}y\)

Short answer

It is verified that \({\left\| {X\hat \beta } \right\|^2} = {\hat \beta ^T}{X^T}{\bf{y}}\) and the standard formula for \(SS\left( E \right)\) is \(SS\left( E \right) = {\left\| {\bf{y}} \right\|^2} - {\hat \beta ^T}{X^T}{\bf{y}}\).

Step-by-step solution

Write the property

1. \({\left\| {\bf{v}} \right\|^2} = {{\bf{v}}^T}{\bf{v}}\)
2. \({\left( {AB} \right)^T} = {B^T}{A^T}\)

Solve \({\left\| {X\hat \beta } \right\|^2}\)

Solve \({\left\| {X\hat \beta } \right\|^2}\) by using the properties \({\left\| {\bf{v}} \right\|^2} &= {{\bf{v}}^T}{\bf{v}}\) and \({\left( {AB} \right)^T} &= {B^T}{A^T}\).

\(\begin{aligned}{\left\| {X\hat \beta } \right\|^2} = {\left( {X\hat \beta } \right)^T}\left( {X\hat \beta } \right)\\ = {{\hat \beta }^T}{X^T}X\hat \beta \end{aligned}\)

Use \({\bf{y}} = X\beta \) in the obtained expression.

\({\left\| {X\hat \beta } \right\|^2} = {\hat \beta ^T}{X^T}{\bf{y}}\)

Hence proved.

Find the value of \(SS\left( E \right)\)

From exercise (19),\(SS\left( T \right) = SS\left( R \right) + SS\left( E \right)\), which implies \(SS\left( E \right) = SS\left( T \right) - SS\left( R \right)\). It is given that \(SS\left( T \right) = {\left\| {\bf{y}} \right\|^2}\) and \(SS\left( R \right) = {\left\| {X\hat \beta } \right\|^2}\).

Substitute the required expressions into \(SS\left( T \right) = SS\left( R \right) + SS\left( E \right)\) and simplify by using \({\left\| {X\hat \beta } \right\|^2} = {\hat \beta ^T}{X^T}{\bf{y}}\).

\(\begin{aligned}SS\left( E \right) = {\left\| {\bf{y}} \right\|^2} - {\left\| {X\hat \beta } \right\|^2}\\ = {\left\| {\bf{y}} \right\|^2} - {{\hat \beta }^T}{X^T}{\bf{y}}\end{aligned}\)

Hence, the standard formula for \(SS\left( E \right)\) is \(SS\left( E \right) = {\left\| {\bf{y}} \right\|^2} - {\hat \beta ^T}{X^T}{\bf{y}}\).