SVD Factorization, Eckart-Young Theorem & PCA
“A = U Sigma V^T, singular values sigma_i = sqrt(lambda_i), Eckart-Young rank-k truncation, and PCA dimensionality reduction.”
1.The Geometry of SVD & Principal Component Analysis
§ 5.1.Singular Value Decomposition (SVD) generalizes eigendecomposition to non-square matrices of arbitrary shape m x n.
Geometric Meaning:
SVD decomposes any linear mapping into three sequential geometric operations: a rotation in the domain V^T, a coordinate stretching along orthogonal axes Sigma, and a rotation in the codomain U.
The Eckart-Young Low-Rank Approximation:
In data science, we frequently compress large datasets by discarding noisy dimensions. The best rank-k approximation in both Frobenius and spectral norm is the truncated SVD:
A_k = Sum_{i=1}^k sigma_i u_i v_i^T.
PCA as SVD:
Given a mean-centered data matrix X (where column means are 0), computing SVD X = U Sigma V^T directly reveals:
1. Principal component loading directions: Columns of V.
2. Low-dimensional projections: Z = X V = U Sigma.
3. Variances along principal axes: lambda_i = sigma_i^2 / (N - 1).
2. Worked Practice Problem: Complete SVD & Rank-1 Approximation
Use u_i = (1 / sigma_i) * A * v_i to ensure the left singular vectors have consistent signs with V.
Problem: Find the singular values, SVD matrices, and the optimal rank-1 approximation A_1 for matrix A = [ [3, 2, 2], [2, 3, -2] ] (size 2 x 3). What fraction of the total variance is captured by A_1?
Step 1: Compute A * A^T (size 2 x 2):
A A^T = [ [3, 2, 2], [2, 3, -2] ] [ [3, 2], [2, 3], [2, -2] ]
Entry (1, 1): 3^2 + 2^2 + 2^2 = 9 + 4 + 4 = 17.
Entry (1, 2): (3)(2) + (2)(3) + (2)(-2) = 6 + 6 - 4 = 8.
Entry (2, 2): 2^2 + 3^2 + (-2)^2 = 4 + 9 + 4 = 17.
A * A^T = [ [17, 8], [8, 17] ].
Step 2: Find eigenvalues of A * A^T:
det([ [17 - lambda, 8], [8, 17 - lambda] ]) = 0
(17 - lambda)^2 - 64 = 0 => 17 - lambda = +- 8.
lambda1 = 25, lambda2 = 9.
Step 3: Extract singular values:
sigma1 = sqrt(25) = 5.
sigma2 = sqrt(9) = 3.
Step 4: Find left singular vectors u1 and u2 (eigenvectors of A A^T):
For lambda1 = 25: (A A^T - 25*I) u1 = 0 => [ [-8, 8], [8, -8] ] u1 = 0 => u1 = [ 1/sqrt(2), 1/sqrt(2) ]^T.
For lambda2 = 9: (A A^T - 9*I) u2 = 0 => [ [8, 8], [8, 8] ] u2 = 0 => u2 = [ 1/sqrt(2), -1/sqrt(2) ]^T.
Step 5: Compute right singular vectors v1 and v2 via v_i = (1 / sigma_i) A^T u_i:
v1 = (1 / 5) [ [3, 2], [2, 3], [2, -2] ] [ 1/sqrt(2), 1/sqrt(2) ]^T
= (1 / (5sqrt(2))) [ 3 + 2, 2 + 3, 2 - 2 ]^T = (1 / (5sqrt(2))) [ 5, 5, 0 ]^T = [ 1/sqrt(2), 1/sqrt(2), 0 ]^T.
v2 = (1 / 3) [ [3, 2], [2, 3], [2, -2] ] [ 1/sqrt(2), -1/sqrt(2) ]^T
= (1 / (3sqrt(2))) [ 3 - 2, 2 - 3, 2 - (-2) ]^T = (1 / (3sqrt(2))) [ 1, -1, 4 ]^T.
Step 6: Compute optimal rank-1 approximation A_1 = sigma1 u1 v1^T:
A_1 = 5 [ 1/sqrt(2), 1/sqrt(2) ]^T [ 1/sqrt(2), 1/sqrt(2), 0 ]
A_1 = 5 (1/2) [ [1, 1, 0], [1, 1, 0] ] = [ [2.5, 2.5, 0], [2.5, 2.5, 0] ].
Step 7: Compute explained variance fraction:
Total variance (Frobenius norm squared): sigma1^2 + sigma2^2 = 25 + 9 = 34.
Variance captured by A_1 = sigma1^2 / (sigma1^2 + sigma2^2) = 25 / 34 ≈ 0.7353 (73.53%).
Conclusion: Singular values are {5, 3}, rank-1 approximation is A_1 = [ [2.5, 2.5, 0], [2.5, 2.5, 0] ], explaining 73.53% of total variance.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.