Question

Let \(X\) be the design matrix in Example 2 corresponding to a least-square fit of parabola to data \(\left( {{x_1},{y_1}} \right), \ldots ,\left( {{x_n},{y_n}} \right)\). Suppose \({x_1}\), \({x_2}\) and \({x_3}\) are distinct. Explain why there is only one parabola that best, in a least-square sense. (See Exercise 5.)

Short answer

The columns of the given matrix are linearly independent, which implies there exists a unique solution of the parabolic equation according to the theorem, so there is only one parabola which the data best.

Step-by-step solution

Theorem

Consider an \(m \times n\) matrix \(A\), the following statements will be logically the same as:

a. For each \({{b}}\) in \({\mathbb{R}^m}\), \(A{\bf{x}} = {\bf{b}}\) has a unique least-squares solution.

b. Columns of the matrix \(A\) are linearly independent.

c. \({A^T}A\) is an invertible matrix.

Show that the given matrix is linearly independent

The matrix in example 2 is,\(X = \left( {\begin{aligned}1&{{x_1}}&{x_1^2}\\1&{{x_2}}&{x_2^2}\\1&{{x_3}}&{x_3^2}\end{aligned}} \right)\).

The matrix columns will be linearly independent if there exist pivot elements in each row of the matrix.

Assume that, \(a = \frac{{{x_2}}}{{{x_1}}}\) and \(b = \frac{{{x_3}}}{{{x_1}}}\), so the matrix becomes:

\(X = \left( {\begin{aligned}1&{{x_1}}&{x_1^2}\\1&{a{x_1}}&{{a^2}x_1^2}\\1&{b{x_1}}&{{b^2}x_1^2}\end{aligned}} \right)\)

So, reduce the matrix in row echelon form to check where three pivot elements exist in rows of the matrix.

\(\left[ {\begin{array}{*{20}} 1&{{x_1}}&{x_1^2} \\ 1&{a{x_1}}&{{a^2}x_1^2} \\ 1&{b{x_1}}&{{b^2}x_1^2} \end{array}} \right]\xrightarrow({{R_3} \to {R_3} - {R_1}}){{{R_2} \to {R_2} - {R_1}}}\left[ {\begin{array}{*{20}} 1&{{x_1}}&{x_1^2} \\ 0&{a{x_1} - {x_1}}&{{a^2}x_1^2 - x_1^2} \\ 0&{b{x_1} - {x_1}}&{{b^2}x_1^2 - x_1^2} \end{array}} \right]\)

\(\left( {\begin{array}{*{20}} 1&{{x_1}}&{x_1^2} \\ 0&{a{x_1} - {x_1}}&{{a^2}x_1^2 - x_1^2} \\ 0&{b{x_1} - {x_1}}&{{b^2}x_1^2 - x_1^2} \end{array}} \right)\xrightarrow({{R_3} \to \frac{{{R_3}}}{{\left( {b - 1} \right){x_1}}}}){{{R_2} \to \frac{{{R_2}}}{{\left( {a - 1} \right){x_1}}}}}\left( {\begin{array}{*{20}} 1&{{x_1}}&{x_1^2} \\ 0&1&{\left( {a + 1} \right){x_1}} \\ 0&1&{\left( {b + 1} \right){x_1}} \end{array}} \right)\)

\(\left( {\begin{array}{*{20}} 1&{{x_1}}&{x_1^2} \\ 0&1&{\left( {a + 1} \right){x_1}} \\ 0&1&{\left( {b + 1} \right){x_1}} \end{array}} \right)\xrightarrow({}){{{R_3} \to {R_3} - {R_2}}}\left( {\begin{array}{*{20}{c}} 1&{{x_1}}&{x_1^2} \\ 0&1&{\left( {a + 1} \right){x_1}} \\ 0&0&{\left( {b - a} \right){x_1}} \end{array}} \right)\)

It can be observed that there are three pivot rows, so columns of the obtained matrix are linearly independent.

Unique solution

From example 2, the corresponding equation of designed matrix is \(y = {\beta _0} + {\beta _1}x + {\beta _2}{x^2}\), which is an equation of parabola.

From step 2, it can be said that the matrix of the corresponding parabola equation is linearly independent, and from the theorem which stated in step 1, it can be said that if the columns of a matrix are linearly independent then a normal equation have a unique solution, so the given parabolic equation has only one parabola which best fits the data.