Question

Find the equation satisfied by the squares of the singular values of the matrix associated with the following over-determined set of equations: $$\begin{array}{rl}2x+3y+z& =0\\ x-y-z& =1\\ 2x+y& =0\\ 2y+z& =-2\end{array}$$ Show that one of the singular values is close to zero. Determine the two larger singular values by an appropriate iteration process and the smallest by indirect calculation.

Short answer

Singular values are square roots of eigenvalues. One is close to zero. Larger values determined iteratively.

Step-by-step solution

Represent the system as a matrix equation

The given system of equations can be written in matrix form as $A\mathbf{x}=\mathbf{b}$, where $A$ is the coefficient matrix, $\mathbf{x}$ is the column vector of the variables, and $\mathbf{b}$ is the column vector of constants. Create the matrices as follows: $A=\left(\begin{array}{ccccccccc}2& 3& 11& -1& -12& 1& 00& 2& 1\end{array}\right)$ and $\mathbf{x}=\left(\begin{array}{c}xyz\end{array}\right)$ and $\mathbf{b}=\left(\begin{array}{c}010-2\end{array}\right)$.

Compute the singular values of matrix A

The singular values of a matrix $A$ are the square roots of the eigenvalues of $A^{T}A$. Calculate $A^{T}A$: $$A^T=\left(\begin{array}{cccccccccc}2& 1& 2& 03& -1& 1& 21& -1& 0& 1\end{array}\right)$$ $$A^{T}A=\left(\begin{array}{ccccccccc}2& 3& 11& -1& -12& 1& 00& 2& 1\end{array}\right)\left(\begin{array}{cccccccccc}2& 1& 2& 03& -1& 1& 21& -1& 0& 1\end{array}\right)=\left(\begin{array}{ccccccc}9& 2& 72& 15& -17& -1& 2\end{array}\right)$$.

Find the eigenvalues of $A^{T}A$

Solve the characteristic equation $det(A^{T}A-\lambda I)=0$ where $I$ is the identity matrix and $\lambda$ are the eigenvalues: $$\left|\begin{array}{ccccccc}9-\lambda & 2& 72& 15-\lambda & -17& -1& 2-\lambda \end{array}\right|=0$$. Expand and solve this equation to find the eigenvalues ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$.

Calculate the singular values

The singular values are the square roots of the eigenvalues calculated in Step 3. $${\sigma }_k=\sqrt{{\lambda }_k}$$. Calculate ${\sigma }_1$, ${\sigma }_2$, and ${\sigma }_3$.

Verify the smallest singular value is close to zero

Check the calculated singular values and determine which one is closest to zero. This confirms one of the singular values is close to zero.

Determine the two larger singular values using iteration

Use an appropriate iterative method (e.g., the power method) to approximate the two larger singular values if not already clear from the direct calculation.

Determine the smallest singular value indirectly

Using the fact one singular value is close to zero, confirm it via indirect methods such as considering the matrix rank-deficiency or verifying orthogonality conditions.

Key concepts

Over-Determined System

An over-determined system is a set of linear equations where there are more equations than unknowns. This can lead to inconsistencies because all equations might not have a common solution. For example, consider the given system of equations. There are four equations (rows) and three unknowns (columns), thus forming an over-determined system. In practice, over-determined systems are common in data fitting and optimization problems, where the goal is to find an approximate solution that minimizes the errors, often handled using least squares methods. Over-determined systems can sometimes be solved by using techniques such as Singular Value Decomposition (SVD) to find the best approximate solution.

Moreover, understanding the properties of over-determined systems can help in identifying when a system is incompatible and what methods can be applied to obtain useful information from it.

Eigenvalues

Eigenvalues are special numbers associated with a matrix, which describe certain properties of the matrix transformations. For a given square matrix A, the eigenvalues are found by solving the characteristic equation $\text{det}(A-\lambda I)=0$, where I is the identity matrix and $\text{det}(...)$ stands for the determinant. In the given exercise, once we have matrix $A^{T}A$, the eigenvalues are computed by finding the roots of its characteristic polynomial. These eigenvalues tell us about the {directional scales} of the transformation represented by the matrix.

For example, in this specific problem, eigenvalues are directly connected to the singular values of the original matrix. The singular values are important in providing insight into the range, null space, and conditioning of the matrix.

Iterative Methods

Iterative methods are mathematical procedures that generate sequences of improving approximate solutions for various types of problems. Unlike direct methods, iterative approaches refine the approximation step by step. In the context of the given problem, an iterative method can be used to find the larger singular values without computing the eigenvalues directly.

A commonly used iterative method for eigenvalues (and thus singular values) is the Power Method, which focuses on finding the largest eigenvalue by repeatedly multiplying a matrix by a vector. This method can be highly efficient, especially for large matrices. Another example includes the QR algorithm, which is well-suited for computing all eigenvalues iteratively.

These methods are especially valuable when dealing with large datasets or matrices, where direct computation becomes impractical.

Matrix Rank

The rank of a matrix denotes the dimension of the vector space generated by its rows or columns. In essence, it tells us the number of linearly independent rows or columns. For the matrix from our exercise, knowing the matrix rank helps in understanding the properties of the system of equations, such as whether the system is consistent or not.

In context of the given problem, the rank is also linked to the singular values. If a singular value is zero, it indicates a deficiency in rank, meaning the matrix does not span its entire vector space. For the given over-determined system, the smallest singular value being close to zero indicates that the matrix is rank-deficient and there's a near dependency among some rows/columns.

Understanding matrix rank helps in various applications like solving linear systems, determining the invertibility of a matrix, and analyzing data.