Dimensionality Reduction: Principal Component Analysis (PCA) & t-SNE
“Covariance matrix diagonalization, maximizing projection variance, eigenvalues, and t-SNE manifolds.”
1.The Two Equivalent Formulations of PCA
§ 6.1.Principal Component Analysis can be formally derived through two complementary mathematical perspectives:
1. Maximum Variance Perspective: Find a unit vector u (u^T u = 1) that maximizes the variance of projected points: max_u u^T Sigma u subject to u^T u = 1. The Lagrangian is L(u, lambda) = u^T Sigma u - lambda (u^T u - 1). Setting derivative to zero yields Sigma u = lambda u (Eigenvalue problem!).
2. Minimum Reconstruction Error Perspective: Find orthogonal subspace that minimizes the sum of squared distances between original points and their low-dimensional projections.
2. Worked Practice Problem: PCA Explained Variance Proportion
Explained variance proportion is the ratio of selected eigenvalues to the trace of the covariance matrix.
Problem: A 3D dataset has sample covariance matrix with eigenvalues lambda1 = 8.0, lambda2 = 3.0, lambda3 = 1.0. What percentage of total variance is retained by projecting onto the top-2 principal components?
Step 1: Compute total variance (Sum of all eigenvalues):
Total Variance = lambda1 + lambda2 + lambda3 = 8.0 + 3.0 + 1.0 = 12.0
Step 2: Compute variance explained by top-2 components:
Top-2 Variance = lambda1 + lambda2 = 8.0 + 3.0 = 11.0
Step 3: Calculate percentage:
Proportion = (11.0 / 12.0) * 100% = 91.67%.
Conclusion: Reducing dimensions from 3D to 2D retains ~91.7% of information while compressing data by 33%.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.