Question

Consider an orthonormal basis$\overrightarrow{{\mathbf{u}}_{\mathbf{1}}}\mathbf{,}\overrightarrow{{\mathbf{u}}_{\mathbf{2}}}\mathbf{,}\mathbf{\dots }\mathbf{,}\overrightarrow{{\mathbf{u}}_{\mathbf{n}}}$ in ${\mathbf{R}}^{\mathbf{m}}$. Find the least square solutions of the system $\mathbf{A}\overrightarrow{\mathbf{x}}\mathbf{=}{\overrightarrow{\mathbf{u}}}_{\mathbf{n}}$where $\mathbf{A}\mathbf{=}\left[\begin{array}{ccc}.& .& .\\ {\overrightarrow{u}}_1& ...& {\overrightarrow{u}}_{n-1}\\ .& .& .\end{array}\right]\mathbf{.}$.

Short answer

The solution is${\mathrm{A}}^{\mathrm{T}}\mathrm{A}{\overrightarrow{\mathrm{x}}}^{\mathrm{k}}={\mathrm{A}}^{\mathrm{T}}{\overrightarrow{\mathrm{u}}}_{\mathrm{n}}$

Step-by-step solution

Step:1 Definition of least square

Consider a linear system as $\mathrm{A}\overrightarrow{\mathrm{x}}={\overrightarrow{\mathrm{u}}}_{\mathrm{n}}$.

Here A is an matrix, a vector$\overrightarrow{\mathrm{x}}$ in${\mathrm{R}}^{\mathrm{n}}$ called a least square solution of this system if$||{\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}{\overrightarrow{\mathrm{x}}}^{\mathrm{k}}||\le ||{\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}\overrightarrow{\mathrm{x}}||$ for all$\overrightarrow{\mathrm{x}}$ in ${\mathrm{R}}^{\mathrm{m}}$.

Step:2 Explanation of the solution

Consider a linear system as $\mathrm{A}\overrightarrow{\mathrm{x}}={\overrightarrow{\mathrm{u}}}_{\mathrm{n}}$where $\left[\begin{array}{ccc}.& .& .\\ {\overrightarrow{\mathrm{u}}}_1& ...& {\overrightarrow{\mathrm{u}}}_{\mathrm{n}-1}\\ .& .& .\end{array}\right]$and A is A is an $\mathrm{n}\times \mathrm{m}$matrix, a vector $\overrightarrow{x}$in ${\mathrm{R}}^{\mathrm{n}}$called a least square solution of this system if $||{\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}{\overrightarrow{\mathrm{x}}}^{\mathrm{k}}||\le ||{\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}\overrightarrow{\mathrm{x}}||$for all $\overrightarrow{\mathrm{x}}$in ${\mathrm{R}}^{\mathrm{m}}$.

The term least square solution reflects the fact that minimizing the sum of the squares of the component of the vector $\left({\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}\overrightarrow{\mathrm{x}}\right)$.

To show ${\overrightarrow{\mathrm{x}}}^{\mathrm{k}}$of a linear system $\mathrm{A}\overrightarrow{\mathrm{x}}={\overrightarrow{\mathrm{u}}}_{\mathrm{n}}$.

Consider the following string of equivalent statements.

The vector ${\overrightarrow{\mathrm{x}}}^{\mathrm{k}}$is a least square solution of the system $\mathrm{A}\overrightarrow{\mathrm{x}}={\overrightarrow{\mathrm{u}}}_{\mathrm{n}}$.

Then by the definition as follows.

$\iff ||{\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}{\overrightarrow{\mathrm{x}}}^{\mathrm{k}}||\le ||{\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}\overrightarrow{\mathrm{x}}||$for all$\overrightarrow{\mathrm{x}}\mathrm{in}{\mathrm{R}}^{\mathrm{m}}$

Also by the theorem 5.4.3 as follows.

$\iff \mathrm{A}\overrightarrow{\mathrm{x}}=\mathrm{proj}\left(\overrightarrow{\mathrm{b}}\right)$where $\mathrm{V}=\mathrm{im}\left(\mathrm{A}\right)$.

Similarly by the theorem 5.1.4 and 5.4.1 as follows.

$\iff {\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}\overrightarrow{\mathrm{x}}$is in ${\mathrm{V}}^{\perp }=(\mathrm{imA}{)}^{\perp }=\mathrm{keR}({\mathrm{A}}^{\mathrm{T}})$

$$\begin{gathered} \Updownarrow \\ {\mathrm{A}}^{\mathrm{T}}({\overrightarrow{\mathrm{u}}}_{\mathrm{n}}-\mathrm{A}\overrightarrow{\mathrm{x}})=\overrightarrow{0} \\ \Updownarrow \\ {\mathrm{A}}^{\mathrm{T}}\mathrm{A}{\overrightarrow{\mathrm{x}}}^{\mathrm{k}}={\mathrm{A}}^{\mathrm{T}}{\overrightarrow{\mathrm{u}}}_{\mathrm{n}} \end{gathered}$$

Therefore, the least square of the system$\mathrm{A}\overrightarrow{\mathrm{x}}={\overrightarrow{\mathrm{u}}}_{\mathrm{n}}$ are the exact solution of the system ${\mathrm{A}}^{\mathrm{T}}\mathrm{A}{\overrightarrow{\mathrm{x}}}^{\mathrm{k}}={\mathrm{A}}^{\mathrm{T}}{\overrightarrow{\mathrm{u}}}_{\mathrm{n}}$.

Thus, the system is called the normal equation of $\mathrm{A}\overrightarrow{\mathrm{x}}={\overrightarrow{\mathrm{u}}}_{\mathrm{n}}$.