Question

By using paper and pencil, find the least squares${\overrightarrow{\mathbf{x}}}^{\mathbf{*}}$ of the system $\mathit{A}\overrightarrow{\mathbf{x}}\mathbf{=}\overrightarrow{\mathbf{b}}$, where$\mathit{A}\mathbf{=}\left[\begin{array}{cc}1& 1\\ 1& 0\\ 1& 1\end{array}\right]$ and$\overrightarrow{\mathbf{b}}\mathbf{=}\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right]$. Verify that the vector$\overrightarrow{\mathbf{b}}\mathbf{-}\mathit{A}{\overrightarrow{\mathbf{x}}}^{\mathbf{*}}$ is perpendicular to the image of A.

Short answer

The least square solution of${\overrightarrow{x}}^{*}$ is$\left[\begin{array}{c}2\\ 2\end{array}\right]$ and the vector$\overrightarrow{b}-A{\overrightarrow{x}}^{*}$ is perpendicular to the image of A .

Step-by-step solution

Determine the least squares solution.

Consider the solution of the equation matrix$A\overrightarrow{x}=\overrightarrow{b}$ where$A=\left[\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right]$ and$\overrightarrow{b}=\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right]$ .

If $\overrightarrow{x}$is the solution of the equation $A\overrightarrow{x}=\overrightarrow{b}$then the least-square solution ${\overrightarrow{x}}^{*}$is defined as${\overrightarrow{x}}^{*}={\left(A^{T}A\right)}^{-1}{A}^T\overrightarrow{b}$ .

Substitute the value $\left[\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right]$for A and$\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right]$ for$\overrightarrow{b}$ in the equation${\overrightarrow{x}}^{*}={\left(A^{T}A\right)}^{-1}{A}^T\overrightarrow{b}$ .

role="math" localid="1660205424631" $$\begin{gathered} {\overrightarrow{x}}^{*}={\left(A^{T}A\right)}^{-1}{A}^T\overrightarrow{b} \\ {\overrightarrow{x}}^{*}={\left({\left(\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right)}^T\left(\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right)\right)}^{-1}{\left(\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right)}^T\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right] \\ {\overrightarrow{x}}^{*}={\left(\left[\begin{array}{ccc}1& 1& 0\\ 1& 0& 1\end{array}\right]\left[\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right]\right)}^{-1}\left[\begin{array}{ccc}1& 1& 0\\ 1& 0& 1\end{array}\right]\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right] \\ {\overrightarrow{x}}^{*}={\left(\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]\right)}^{-1}\left[\begin{array}{c}6\\ 6\end{array}\right] \end{gathered}$$

Further, simplify the equation as follows.

$$\begin{gathered} {\overrightarrow{x}}^{*}={\left(\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]\right)}^{-1}\left[\begin{array}{c}6\\ 6\end{array}\right] \\ {\overrightarrow{x}}^{*}={\left(\left[\begin{array}{cc}2/3& 1/3\\ -1/3& 2/3\end{array}\right]\right)}^{-1}\left[\begin{array}{c}6\\ 6\end{array}\right] \\ {\overrightarrow{x}}^{*}=\left[\begin{array}{c}2\\ 2\end{array}\right] \end{gathered}$$

Therefore, the least square solution of ${\overrightarrow{x}}^{*}$is$\left[\begin{array}{c}2\\ 2\end{array}\right]$ .

Show that explain the meaning of b→-Ax→* is perpendicular to the image of A .

The image of the matrix A is defined as follows.

$$\begin{gathered} lm\left(A\right)=\left\{A\overline{)x}x\in {\mathrm{ℝ}}^2\right\} \\ =\left\{\left[\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right]\left[\begin{array}{c}a\\ b\end{array}\right]\overline{)x}\in {\mathrm{ℝ}}^2\right\} \\ lm\left(A\right)=\left\{\left[\begin{array}{c}a+b\\ a\\ b\end{array}\right]\overline{)x}\in {\mathrm{ℝ}}^2\right\} \end{gathered}$$

Substitute the value $\left[\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right]$for A , $\left[\begin{array}{c}a+b\\ a\\ b\end{array}\right]$for $lm\left(A\right)$, $\left[\begin{array}{c}2\\ 2\end{array}\right]$for ${\overrightarrow{x}}^{*}$and $\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right]$for $\overrightarrow{b}$in role="math" localid="1660211257336" ${\left[\overrightarrow{b}-A{\overrightarrow{x}}^{*}\right]}^{T}lm\left(A\right)$as follows.

$$\begin{gathered} {\left(\overrightarrow{b}-A{\overrightarrow{x}}^{*}\right)}^{T}lm\left(A\right)={\left[\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right]-\left[\begin{array}{cc}1& 1\\ 1& 0\\ 0& 1\end{array}\right]\left[\begin{array}{c}2\\ 2\end{array}\right]\right]}^T\left[\begin{array}{c}a+b\\ a\\ b\end{array}\right] \\ ={\left[\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right]-\left[\begin{array}{c}4\\ 2\\ 2\end{array}\right]\right]}^T\left[\begin{array}{c}a+b\\ a\\ b\end{array}\right] \\ ={\left[\left[\begin{array}{c}-1\\ 1\\ 1\end{array}\right]\right]}^T\left[\begin{array}{c}a+b\\ a\\ b\end{array}\right] \\ {\left[\overrightarrow{b}-A{\overrightarrow{x}}^{*}\right]}^{T}lm\left(A\right)=\left[\begin{array}{ccc}-1& 1& 1\end{array}\right]\left[\begin{array}{c}a+b\\ a\\ b\end{array}\right] \end{gathered}$$

Further, simplify the equation as follows.

$$\begin{gathered} {\left[\overrightarrow{b}-A{\overrightarrow{x}}^{*}\right]}^{T}lm\left(A\right)=\left[\begin{array}{ccc}-1& 1& 1\end{array}\right]\left[\begin{array}{c}a+b\\ a\\ b\end{array}\right] \\ =-a-b+a+b \\ {\left[\overrightarrow{b}-A{\overrightarrow{x}}^{*}\right]}^{T}lm\left(A\right)=0 \end{gathered}$$

By the definition of perpendicular of two vectors, the vector$\overrightarrow{b}-A{\overrightarrow{x}}^{*}$ is perpendicular to the image of A .

If${\overrightarrow{x}}^{*}=0$ then it implies$A{\overrightarrow{x}}^{*}=\overrightarrow{0}$ .

Hence, the least square solution of ${\overrightarrow{x}}^{*}$is $\left[\begin{array}{c}2\\ 2\end{array}\right]$and the vector $\overrightarrow{b}-A{\overrightarrow{x}}^{*}$is perpendicular to the image of A .