Ridge (L2) vs Lasso (L1) Regularization & Geometric Sparsity
“Shrinkage mechanics, solving ill-conditioned matrices, L1 subgradients, and feature selection.”
1.The Mathematical Duality Between L1 and L2 Norm Penalties
§ 1.1.When features are collinear, the eigenvalues of X^T X approach zero, causing OLS weight variance to explode (Var(w*) = sigma^2 (X^T X)^{-1}). Regularization restores numerical stability by penalizing model complexity.
Ridge Regression (L2):
By adding lambda * I to X^T X, every eigenvalue is shifted upwards by lambda (lambda_i(X^T X + lambda I) = lambda_i(X^T X) + lambda > 0). Hence, the regularized matrix is strictly invertible even when columns are linearly dependent.
Why Lasso Produces Exact Sparsity:
The Lasso constraint region is a rotated hypercube (Sum |w_i| <= C), which features sharp corners on the coordinate axes. The elliptical contours of the quadratic RSS loss touch these corners first with high probability, setting entire dimensions to exact zero and performing embedded feature selection.
2. Worked Practice Problem: Ridge Shrinkage Computation
Ensure lambda is added along the diagonal of X^T X, not added to individual elements.
Problem: An orthonormal design matrix X satisfies X^T X = I. If the OLS estimate is w_{ols} = [4, -2]^T and Ridge regularization parameter is lambda = 1, find the Ridge weight vector w_{ridge}.
Step 1: Write Ridge closed-form formula for orthonormal features:
w_{ridge} = (X^T X + lambda * I)^{-1} X^T y
Step 2: Since X^T y = (X^T X) w_{ols} = I * w_{ols} = w_{ols}:
w_{ridge} = (I + lambda I)^{-1} w_{ols} = [ (1 + lambda) I ]^{-1} w_{ols} = [ 1 / (1 + lambda) ] w_{ols}
Step 3: Substitute lambda = 1:
w_{ridge} = [ 1 / (1 + 1) ] [ 4, -2 ]^T = (1/2) [ 4, -2 ]^T = [ 2, -1 ]^T.
Conclusion: Under orthonormal features, Ridge shrinks every parameter uniformly by a factor of 1 / (1 + lambda).
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.