Exercises: SVD and Low-Rank Approximation

Practice problems and worked solutions for singular values, truncated SVD, and rank-k approximation.

Keywords

exercises, svd, low-rank approximation

1 Scope and Goals

This set trains four things:

• reading the singular structure of simple matrices
• building low-rank approximations by hand when the SVD is simple
• connecting singular values to norm calculations
• translating the same ideas into PCA and numerical approximation language

2 Prerequisites

• orthogonality and orthonormal bases
• eigenvalues of symmetric matrices
• Frobenius norm and matrix rank
• the definitions from SVD and Low-Rank Approximation

3 Warm-Up Problems

1. Let

   \[ A = \begin{bmatrix} 5 & 0 \\ 0 & 2 \end{bmatrix}. \]

   What are its singular values?

2. Let

   \[ B = \begin{bmatrix} 4 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}. \]

   Write the best rank-\(1\) approximation \(B_1\).

4 Core Problems

1. For

   \[ A = \begin{bmatrix} 3 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}, \]

   write a thin SVD and compute the best rank-\(1\) approximation.

2. Show that if \(A\) has orthonormal columns, then every singular value of \(A\) equals \(1\).

3. Let

   \[ A = \operatorname{diag}(4,2,1). \]

   Compute \(\|A-A_1\|_F\), \(\|A-A_2\|_F\), and \(\|A-A_1\|_2\).

5 Proof Problems

1. Show that if \(A = U_r \Sigma_r V_r^\top\), then

   \[ A^\top A = V_r \Sigma_r^2 V_r^\top. \]

2. Prove that

   \[ \|A\|_F^2 = \sum_{i=1}^r \sigma_i^2. \]

6 Computational or Applied Problems

1. Open Computation Lab: Rank-1 Approximation and PCA Geometry. Set the retained rank to 1 and vary the noise scale. Record how the second singular value and the Frobenius reconstruction error change together.
2. In software, take a centered \(m \times 2\) data matrix and compute its SVD. Verify numerically that the top right singular vector matches the first principal direction of \(X^\top X\).

7 Hints

1. For diagonal matrices, the singular values are the absolute values of the diagonal entries.
2. The best low-rank approximation of a diagonal matrix keeps the largest diagonal entries and zeros the rest.
3. If the columns are orthonormal, then \(A^\top A = I\).
4. Use \(\|A\|_F^2 = \operatorname{trace}(A^\top A)\).

8 Full Solutions

8.1 Solution 1

For

\[ A = \begin{bmatrix} 5 & 0 \\ 0 & 2 \end{bmatrix}, \]

the singular values are \(5\) and \(2\).

8.2 Solution 2

The singular values are \(4\) and \(1\), so the best rank-\(1\) approximation keeps only the top singular direction:

\[ B_1 = \begin{bmatrix} 4 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}. \]

8.3 Solution 3

A thin SVD is

\[ A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}. \]

So the best rank-\(1\) approximation is

\[ A_1 = \begin{bmatrix} 3 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}. \]

8.4 Solution 4

If \(A\) has orthonormal columns, then

\[ A^\top A = I. \]

So the eigenvalues of \(A^\top A\) are all \(1\), and the singular values, being their square roots, are all \(1\).

8.5 Solution 5

For \(A = \operatorname{diag}(4,2,1)\),

\[ \|A-A_1\|_F = \sqrt{2^2+1^2} = \sqrt{5}, \qquad \|A-A_2\|_F = 1, \]

and

\[ \|A-A_1\|_2 = 2. \]

8.6 Solution 6

Starting from \(A = U_r \Sigma_r V_r^\top\),

\[ A^\top A = V_r \Sigma_r U_r^\top U_r \Sigma_r V_r^\top = V_r \Sigma_r^2 V_r^\top, \]

because \(U_r^\top U_r = I\).

8.7 Solution 7

Using the SVD,

\[ A^\top A = V_r \Sigma_r^2 V_r^\top. \]

Taking the trace and using cyclic invariance,

\[ \|A\|_F^2 = \operatorname{trace}(A^\top A) = \operatorname{trace}(V_r \Sigma_r^2 V_r^\top) = \operatorname{trace}(\Sigma_r^2) = \sum_{i=1}^r \sigma_i^2. \]

8.8 Solution 8

In the lab, when the retained rank is \(1\), the reconstruction error in Frobenius norm tracks the discarded singular value in this two-feature setting. As the noise scale increases, the second singular value grows, and so does the rank-\(1\) reconstruction error.

The point is not only numerical. It is a direct visualization of the Eckart-Young theorem in a small data matrix.

8.9 Solution 9

If \(X = U \Sigma V^\top\), then

\[ X^\top X = V \Sigma^2 V^\top. \]

So the top eigenvector of \(X^\top X\) is the first right singular vector of \(X\).

Numerically, software should return the same direction up to sign. If one method returns \(v\) and the other \(-v\), they still represent the same principal component.

9 Sources and Further Reading

• MIT 18.06 Lecture 29: Singular value decomposition - First pass - good official problem source for basic SVD fluency. Checked 2026-04-24.
• Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares - Second pass - useful once you want computational interpretation. Checked 2026-04-24.
• Hefferon, Linear Algebra - Second pass - additional exercises and proof reinforcement. Checked 2026-04-24.
• CS168 Lecture 9: The Singular Value Decomposition and Low-Rank Matrix Approximations - Paper bridge - useful for seeing the same exercises turn into application stories. Checked 2026-04-24.