Question

Given data for a least-squares problem, \(\left( {{x_1},{y_1}} \right), \ldots ,\left( {{x_n},{y_n}} \right)\), the following abbreviations are helpful:

\(\begin{aligned}{l}\sum x = \sum\nolimits_{i = 1}^n {{x_i}} ,{\rm{ }}\sum {{x^2}} = \sum\nolimits_{i = 1}^n {x_i^2} ,\\\sum y = \sum\nolimits_{i = 1}^n {{y_i}} ,{\rm{ }}\sum {xy} = \sum\nolimits_{i = 1}^n {{x_i}{y_i}} \end{aligned}\)

The normal equations for a least-squares line \(y = {\hat \beta _0} + {\hat \beta _1}x\) may be written in the form

\(\begin{aligned}{c}{{\hat \beta }_0} + {{\hat \beta }_1}\sum x = \sum y \\{{\hat \beta }_0}\sum x + {{\hat \beta }_1}\sum {{x^2}} = \sum {xy} {\rm{ (7)}}\end{aligned}\)

Derive the normal equations (7) from the matrix form given in this section.

Short answer

The normal equations (7) are derived by using given data points.

Step-by-step solution

The General Linear Model

The equation of the general linear model is defined as:

\({\bf{y}} = X\beta + \in \)

Where, \({\bf{y}} = \left( {\begin{aligned}{{y_1}}\\{{y_2}}\\ \vdots \\{{y_n}}\end{aligned}} \right)\) is an observational vector, \(X = \left( {\begin{aligned}1&{{x_1}}& \cdots &{x_1^n}\\1&{{x_2}}& \cdots &{x_2^n}\\ \vdots & \vdots & \ddots & \vdots \\1&{{x_n}}& \cdots &{x_n^n}\end{aligned}} \right)\) is the design matrix, \(\beta = \left( {\begin{aligned}{{\beta _1}}\\{{\beta _2}}\\ \vdots \\{{\beta _n}}\end{aligned}} \right)\) is the parameter vector, and \( \in = \left( {\begin{aligned}{{ \in _1}}\\{{ \in _2}}\\ \vdots \\{{ \in _n}}\end{aligned}} \right)\) is the residual vector.

Find design matrix, observation vector, parameter vector for given data

The given data points are\(\left( {{x_1},{y_1}} \right), \ldots ,\left( {{x_n},{y_n}} \right)\).

Write the design matrix and observational vector for the given data points.

Design matrix: \(X = \left( {\begin{aligned}1&{{x_1}}\\1&{{x_2}}\\ \vdots & \vdots \\1&{{x_n}}\end{aligned}} \right)\)

Observational matrix: \({\bf{y}} = \left( {\begin{aligned}{{y_1}}\\{{y_2}}\\ \vdots \\{{y_n}}\end{aligned}} \right)\)

And the parameter vectorfor the given equation is,

\({\bf{\beta }} = \left( {\begin{aligned}{{\beta _0}}\\{{\beta _1}}\end{aligned}} \right)\)

Normal equation

The normal equation is given by,

\({X^T}X\beta = {X^T}{\bf{y}}\)

Derive the normal equation

As the normal equation is\({X^T}X\beta = {X^T}{\bf{y}}\), by using this equation, derive equations (7) for the given data points.

Find \({X^T}X\) and \({X^T}{\bf{y}}\).

\(\begin{aligned}{X^T}X &= {\left( {\begin{aligned}1&{{x_1}}\\1&{{x_2}}\\ \vdots & \vdots \\1&{{x_n}}\end{aligned}} \right)^T}\left( {\begin{aligned}1&{{x_1}}\\1&{{x_2}}\\ \vdots & \vdots \\1&{{x_n}}\end{aligned}} \right)\\ &= \left( {\begin{aligned}{1 + 1 + \cdots + n{\rm{ times}}}&{{x_1} + \cdots + {x_n}}\\{{x_1} + \cdots + {x_n}}&{{x_1}{x_1} + \cdots + {x_n}{x_n}}\end{aligned}} \right)\\ &= \left( {\begin{aligned}n&{\sum x }\\{\sum x }&{\sum {{x^2}} }\end{aligned}} \right)\end{aligned}\)

\(\begin{aligned}{X^T}{\bf{y}} &= {\left( {\begin{aligned}1&{{x_1}}\\1&{{x_2}}\\ \vdots & \vdots \\1&{{x_n}}\end{aligned}} \right)^T}\left( {\begin{aligned}{{y_1}}\\{{y_2}}\\ \vdots \\{{y_n}}\end{aligned}} \right)\\ &= \left( {\begin{aligned}{{y_1} + \cdots + {y_n}}\\{{x_1}{y_1} + \cdots + {x_n}{y_n}}\end{aligned}} \right)\\ &= \left( {\begin{aligned}{\sum y }\\{\sum {xy} }\end{aligned}} \right)\end{aligned}\)

Then, the equation \({X^T}X\beta = {X^T}{\bf{y}}\) becomes:

\(\begin{aligned}\left( {\begin{aligned}n&{\sum x }\\{\sum x }&{\sum {{x^2}} }\end{aligned}} \right)\left( {\begin{aligned}{{{\hat \beta }_0}}\\{{{\hat \beta }_1}}\end{aligned}} \right) &= \left( {\begin{aligned}{\sum y }\\{\sum {xy} }\end{aligned}} \right)\\\left( {\begin{aligned}{n{{\hat \beta }_0} + {{\hat \beta }_1}\sum x }\\{{{\hat \beta }_0}\sum x + {{\hat \beta }_1}\sum {{x^2}} }\end{aligned}} \right) &= \left( {\begin{aligned}{\sum y }\\{\sum {xy} }\end{aligned}} \right)\end{aligned}\)

Write the obtained system in the form of equations.

\(\begin{aligned}n{{\hat \beta }_0} + {{\hat \beta }_1}\sum x = \sum y \\{{\hat \beta }_0}\sum x + {{\hat \beta }_1}\sum {{x^2}} = \sum {xy} \end{aligned}\)

These are the required equations.