Question

Least Squares Given \(n\) points \(\left(x_{1}, y_{1}\right),\left(x_{2}, y_{2}\right), \ldots\), \(\left(x_{n}, y_{n}\right)\) in the \(x y\) -plane, we wish to find the line \(y=m x+b\) such that the sum of the squares of the vertical distances from the points to the line is a minimum; that is, we wish to minimize $$ f(m, b)=\sum_{i=1}^{n}\left(y_{i}-m x_{i}-b\right)^{2} $$ (a) Find \(\partial f / \partial m\) and \(\partial f / \partial b\), and set these results equal to zero. Show that this leads to the system of equations $$ \begin{aligned} m \sum_{i=1}^{n} x_{i}^{2}+b \sum_{i=1}^{n} x_{i} &=\sum_{i=1}^{n} x_{i} y_{i} \\\ m \sum_{i=1}^{n} x_{i}+n b &=\sum_{i=1}^{n} y_{i} \end{aligned} $$ (b) Solve this system for \(m\) and \(b\). (c) Use the Second Partials Test (Theorem C) to show that \(f\) is minimized for this choice of \(m\) and \(b\).

Short answer

The least squares solution is \( m = \frac{ n \sum x_i y_i - \sum x_i \sum y_i }{n \sum x_i^2 - (\sum x_i)^2} \) and \( b = \frac{ \sum y_i - m \sum x_i }{n} \), verified by the second partials test for a minimum.

Step-by-step solution

Differentiate with respect to m

We start by finding the partial derivative of the function \( f(m, b) = \sum_{i=1}^{n} (y_i - mx_i - b)^2 \) with respect to \( m \). This derivative is obtained as follows: \[ \frac{\partial f}{\partial m} = \sum_{i=1}^{n} -2x_i(y_i - mx_i - b). \] Using the linearity of the sum, this expression simplifies to: \[ -2 \sum_{i=1}^{n} x_i(y_i - mx_i - b). \] Setting this equal to zero, we get one equation in terms of \( m \) and \( b \).

Differentiate with respect to b

Next, we find the partial derivative of \( f(m, b) \) with respect to \( b \): \[ \frac{\partial f}{\partial b} = \sum_{i=1}^{n} -2(y_i - mx_i - b). \] Simplifying, we have: \[ -2 \sum_{i=1}^{n} (y_i - mx_i - b). \] Setting this equal to zero gives us the second equation in terms of \( m \) and \( b \).

Setting Partial Derivatives to Zero

By setting \( \frac{\partial f}{\partial m} = 0 \) and \( \frac{\partial f}{\partial b} = 0 \), we obtain the system of equations: \[ m \sum_{i=1}^{n} x_{i}^{2} + b \sum_{i=1}^{n} x_{i} = \sum_{i=1}^{n} x_{i} y_{i} \] \[ m \sum_{i=1}^{n} x_{i} + nb = \sum_{i=1}^{n} y_{i}. \] These represent the conditions for the least squares fit line.

Solve for m and b

Solve the system of linear equations obtained in Step 3. First, solve the system by expressing it in matrix form and using methods like substitution or matrix inversion. By solving the system, you find the values of \( m \) and \( b \) that minimize \( f(m, b) \). This provides the equations used to calculate \( m \) and \( b \):\[ m = \frac{ n \sum x_i y_i - \sum x_i \sum y_i }{n \sum x_i^2 - (\sum x_i)^2} \]\[ b = \frac{ \sum y_i - m \sum x_i }{n}. \]

Verify Minimization with Second Partials Test

To confirm that these values minimize \( f \), apply the Second Partials Test. This involves computing the second partial derivatives:\[ \frac{\partial^2 f}{\partial m^2} = 2 \sum x_i^2, \quad \frac{\partial^2 f}{\partial b^2} = 2n, \quad \text{and} \quad \frac{\partial^2 f}{\partial m \partial b} = 2 \sum x_i. \]Use these to check the determinant of the Hessian matrix:\[ H = \begin{bmatrix} \frac{\partial^2 f}{\partial m^2} & \frac{\partial^2 f}{\partial m \partial b} \ \frac{\partial^2 f}{\partial m \partial b} & \frac{\partial^2 f}{\partial b^2} \end{bmatrix}. \]Verify that the determinant is positive and \( \frac{\partial^2 f}{\partial m^2} > 0 \), confirming a minimum.

Key concepts

Partial Derivatives

The concept of partial derivatives is essential when working with functions of multiple variables. A partial derivative measures how the function changes as one variable changes, while the other variables are kept constant. This is particularly useful in optimization problems where you want to understand how each variable influences the function independently.

In the Least Squares method, we aim to find values of \( m \) and \( b \) that minimize the error function \( f(m, b) = \sum_{i=1}^{n} (y_i - mx_i - b)^2 \). This function tells us how well a line fits a set of points. By computing the partial derivatives \( \frac{\partial f}{\partial m} \) and \( \frac{\partial f}{\partial b} \), we pinpoint where the slope and intercept contribute to reducing the error.

To compute these derivatives, we use the rules of differentiation:

• For \( \frac{\partial f}{\partial m} \), consider the derivative of \((y_i - mx_i - b)^2\) with respect to \( m \).
• For \( \frac{\partial f}{\partial b} \), consider the derivative of \((y_i - mx_i - b)^2\) with respect to \( b \).

Setting these equations to zero determines where the error is minimized concerning \( m \) and \( b \). This leads us to the next concept, the creation of a system of equations.

System of Equations

Once we have the partial derivatives, \( \frac{\partial f}{\partial m} = 0 \) and \( \frac{\partial f}{\partial b} = 0 \), we can form a system of linear equations. Solving this system helps us find the optimal values for \( m \) and \( b \) that minimize the sum of squared errors.

This is represented by two equations involving summations over the dataset, capturing the influence of all data points:
- \( m \sum_{i=1}^{n} x_{i}^{2} + b \sum_{i=1}^{n} x_{i} = \sum_{i=1}^{n} x_{i} y_{i} \)
- \( m \sum_{i=1}^{n} x_{i} + nb = \sum_{i=1}^{n} y_{i} \)

These equations can be thought of as encapsulating the balance of the data regarding the line that best fits the points. Solving them often involves methods like substitution or matrix inversion, especially useful when dealing with a larger number of points. In this context, the solutions represent the line of best fit in terms of \( m \) and \( b \).

Understanding and solving systems of equations is crucial in numerous fields. They help in finding relationships between variables, making predictions, and optimizing results.

Second Partials Test

Verification that a minimum has been found involves more than just finding \( m \) and \( b \). The Second Partials Test is applied to ensure that the solution is indeed a minimum and not a maximum or a saddle point.

For a function \( f(m, b) \), the second partial derivatives provide further insight into how the function behaves around an optimal point:

• \( \frac{\partial^2 f}{\partial m^2} = 2 \sum x_i^2 \)
• \( \frac{\partial^2 f}{\partial b^2} = 2n \)
• \( \frac{\partial^2 f}{\partial m \partial b} = 2 \sum x_i \)

By arranging these in the Hessian matrix \( H \), we compute the determinant:
\[ H = \begin{bmatrix} \frac{\partial^2 f}{\partial m^2} & \frac{\partial^2 f}{\partial m \partial b} \ \frac{\partial^2 f}{\partial m \partial b} & \frac{\partial^2 f}{\partial b^2} \end{bmatrix}. \]
The test confirms a minimum if the determinant is positive and \( \frac{\partial^2 f}{\partial m^2} \) is also positive.

This test is valuable for ensuring that our least squares solution is not only mathematically convenient but also logically sound in terms of minimizing error.