Symmetric Matrices, Spectral Theorem & Positive Definiteness
“Real eigenvalues, orthogonal diagonalization A = Q Lambda Q^T, quadratic forms x^T A x, and Sylvester's criterion.”
1.The Spectral Theorem & Quadratic Forms in Machine Learning
§ 4.1.Real symmetric matrices A = A^T are the cornerstone of multivariate statistics, Hessian curvature optimization, and kernel methods.
The Spectral Theorem:
For any real symmetric matrix A:
1. All eigenvalues lambda_1, ..., lambda_n are real.
2. Eigenvectors can be chosen to form an orthonormal basis of R^n.
3. A = Q Lambda Q^T, where Q is an orthogonal matrix (Q^T Q = I).
Classification of Quadratic Forms q(x) = x^T A x:
- Positive Definite (PD): x^T A x > 0 for all x != 0. (Strict local minimum in optimization).
- Positive Semi-Definite (PSD): x^T A x >= 0 for all x. (Convex optimization landscape; Covariance matrices are always PSD).
- Negative Definite (ND): x^T A x < 0 for all x != 0. (Strict local maximum).
- Indefinite: x^T A x takes both positive and negative values. (Saddle point).
2. Worked Practice Problem: Sylvester's Criterion & Spectral Decomposition
Confirm all leading principal minors are strictly positive for strict positive definiteness.
Problem: Let A = [ [2, -1, 0], [-1, 2, -1], [0, -1, 2] ]. (a) Test if A is Positive Definite using Sylvester's criterion. (b) Find its eigenvalues to confirm.
Step 1: Compute leading principal minors (Sylvester's Criterion):
D1 = det([ 2 ]) = 2 > 0.
D2 = det([ [2, -1], [-1, 2] ]) = (2)(2) - (-1)(-1) = 4 - 1 = 3 > 0.
D3 = det(A) = expand along row 1:
D3 = 2 det([ [2, -1], [-1, 2] ]) - (-1) det([ [-1, -1], [0, 2] ]) + 0
D3 = 2 (3) + 1 (-2 - 0) = 6 - 2 = 4 > 0.
All leading principal minors (D1 = 2, D2 = 3, D3 = 4) are strictly positive.
Therefore, A is strictly POSITIVE DEFINITE (PD).
Step 2: Solve characteristic polynomial det(A - lambda * I) = 0:
det([ [2 - lambda, -1, 0], [-1, 2 - lambda, -1], [0, -1, 2 - lambda] ]) = 0
(2 - lambda) [ (2 - lambda)^2 - 1 ] - (-1) [ -(2 - lambda) ] = 0
(2 - lambda) * [ (2 - lambda)^2 - 1 ] - (2 - lambda) = 0
(2 - lambda) * [ (2 - lambda)^2 - 2 ] = 0.
Step 3: Extract eigenvalues:
lambda1 = 2
(2 - lambda)^2 = 2 => 2 - lambda = +- sqrt(2)
lambda2 = 2 - sqrt(2) ≈ 0.586
lambda3 = 2 + sqrt(2) ≈ 3.414.
Step 4: Verify definiteness via eigenvalues:
All three eigenvalues (0.586, 2, 3.414) are strictly positive real numbers (> 0).
Conclusion: A is positive definite by both Sylvester's criterion and positive eigenvalues.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.