Linear Discriminant Analysis (LDA) & Fisher's Criterion
“Maximizing between-class separation while minimizing within-class variance, Fisher's ratio, and Bayes optimal boundaries.”
1.Theoretical Derivation of Fisher's Linear Discriminant
§ 2.1.While PCA finds directions of maximum variance regardless of class labels, Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction technique and linear classifier originally formulated by Ronald Fisher in 1936.
Fisher's Objective Criterion:
Consider projecting d-dimensional data vector x onto a 1-dimensional line via scalar projection: y = w^T x. Let mu_1 and mu_2 be the class sample means. The projected means are m_1 = w^T mu_1 and m_2 = w^T mu_2.
The difference between projected means squared is: (m_1 - m_2)^2 = (w^T (mu_1 - mu_2))^2 = w^T (mu_1 - mu_2)(mu_1 - mu_2)^T w = w^T S_B w, where S_B = (mu_1 - mu_2)(mu_1 - mu_2)^T is the Between-Class Scatter Matrix.
The projected within-class scatter is: s_1^2 + s_2^2 = Sum_{x in C_1} (w^T x - w^T mu_1)^2 + Sum_{x in C_2} (w^T x - w^T mu_2)^2 = w^T S_W w, where S_W = S_1 + S_2 is the Within-Class Scatter Matrix.
Fisher's criterion maximizes the ratio of between-class scatter to within-class scatter: J(w) = (w^T S_B w) / (w^T S_W w).
Solving for the Optimal Projection Vector w*:
Differentiating J(w) with respect to w and setting to zero yields the generalized eigenvalue problem: S_B w = lambda S_W w.
Notice that S_B w = (mu_1 - mu_2)(mu_1 - mu_2)^T w. The term (mu_1 - mu_2)^T w is simply a scalar alpha. Thus, S_B w = alpha (mu_1 - mu_2) always points in the direction of the mean difference vector!
Substituting this into the eigenvalue equation: lambda S_W w = alpha (mu_1 - mu_2) => w* = S_W^{-1} (mu_1 - mu_2) (ignoring unimportant scalar scaling factor alpha / lambda).
Under the probabilistic generative perspective, if each class c follows a multivariate normal distribution N(mu_c, Sigma) with a shared covariance matrix Sigma, the log-posterior ratio ln(P(C_1|x) / P(C_2|x)) is strictly linear in x with weight vector w = Sigma^{-1} (mu_1 - mu_2). If class covariances differ (Sigma_1 != Sigma_2), the quadratic term x^T Sigma_c^{-1} x does not cancel, yielding Quadratic Discriminant Analysis (QDA).
2. Worked Practice Problem: 2D Fisher's Linear Discriminant Derivation
Do not invert individual class matrices S_1 and S_2. You must sum them first to form S_W = S_1 + S_2 before inverting.
Problem: A 2D binary classification problem has two classes C1 and C2 with sample means: mu_1 = [1, 2]^T and mu_2 = [3, 4]^T. The class scatter matrices are: S_1 = [ 2 1 ] / [ 1 2 ] and S_2 = [ 1 0 ] / [ 0 1 ]. Compute the optimal Fisher linear projection vector w*.
Step 1: Compute the mean difference vector (mu_1 - mu_2):
mu_1 - mu_2 = [ 1 - 3, 2 - 4 ]^T = [ -2, -2 ]^T
Step 2: Compute Within-Class Scatter Matrix S_W = S_1 + S_2:
S_W = [ 2 1 ] + [ 1 0 ] = [ 3 1 ]
[ 1 2 ] [ 0 1 ] [ 1 3 ]
Step 3: Invert S_W:
Determinant Det(S_W) = (3)(3) - (1)(1) = 9 - 1 = 8.
S_W^{-1} = (1/8) * [ 3 -1 ]
[ -1 3 ]
Step 4: Compute w* = S_W^{-1} (mu_1 - mu_2):
w = (1/8) [ 3 -1 ] [ -2 ] = (1/8) [ 3(-2) + (-1)(-2) ] = (1/8) [ -6 + 2 ] = (1/8) * [ -4 ] = [ -1/2 ]
[ -1 3 ] [ -2 ] [ (-1)(-2) + 3(-2) ] [ 2 - 6 ] [ -4 ] [ -1/2 ]
Step 5: Simplify direction:
w* is proportional to [ 1, 1 ]^T or [ -1, -1 ]^T.
Conclusion: The optimal 1D line that maximizes class separability is y = x_1 + x_2. Projecting data onto w* collapses 2D coordinates into a 1D scalar where class overlap is minimized.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.