Question

Least Squares Regression Line Given \(n\) points \(\left(x_{1}, y_{1}\right),\) \(\left(x_{2}, y_{2}\right), \ldots,\left(x_{n}, y_{n}\right),\) where the \(x_{i}\) 's are not all alike, the least squares regression line is the line \(y=a x+b\) that minimizes the sum of the squares of the vertical distances from the points to the line. Use calculus to show that $$ S(a, b)=\sum_{i=1}^{n}\left(a x_{i}+b-y_{i}\right)^{2} $$ is minimum when \(a\) and \(b\) are the unique solutions to the system of equations $$ \begin{aligned} n b+\left(\sum_{i=1}^{n} x_{i}\right) a &=\sum_{i=1}^{n} y_{i} \\ \left(\sum_{i=1}^{n} x_{i}\right) b+\left(\sum_{i=1}^{n} x_{i}^{2}\right) a &=\sum_{i=1}^{n} x_{i} y_{i} \end{aligned} $$

Short answer

The least squares coefficients \(a\) and \(b\) are obtained by solving the system of two equations: \(n b+\left(\sum_{i=1}^{n} x_{i}\right) a =\sum_{i=1}^{n} y_{i}\) and \(\left(\sum_{i=1}^{n} x_{i}\right) b+\left(\sum_{i=1}^{n} x_{i}^{2}\right) a =\sum_{i=1}^{n} x_{i} y_{i} \).

Step-by-step solution

Expression for Squared Residuals

Start by writing an expression for the residuals. The residual for each point \(i\) is the difference between the observed \(y\) value and the predicted \(y\) value (which is \(ax_i + b\)). Square these residuals and sum them over all \(n\) points to get \( S(a, b)=\sum_{i=1}^{n}\left(a x_{i}+b-y_{i}\right)^{2} \).

Deriving the Normal Equations

We need to minimize \( S(a, b) \). That means we need to take the derivative of \( S(a, b) \) with respect to \(a\) and \(b\), and set each to 0. This results in a system of two equations, also known as the Normal Equations.

Derive First Equation

Take the derivative of \( S(a, b) \) with respect to \(b\), set it to 0, and simplify to get the first equation: \(n b+\left(\sum_{i=1}^{n} x_{i}\right) a =\sum_{i=1}^{n} y_{i}\).

Derive Second Equation

Take the derivative of \( S(a, b) \) with respect to \(a\), set it to 0, and simplify to get the second equation: \(\left(\sum_{i=1}^{n} x_{i}\right) b+\left(\sum_{i=1}^{n} x_{i}^{2}\right) a =\sum_{i=1}^{n} x_{i} y_{i} \).