Point Estimation: Maximum Likelihood Estimation (MLE)
“Likelihood functions, log-likelihood optimization, score function, Hessian concavity, and estimator properties.”
1.The Principle of Maximum Likelihood
§ 4.1.Given observed data x_1, x_2, ..., x_n drawn i.i.d. from a parametric distribution with parameter vector theta, point estimation seeks the single optimal numerical estimate theta_hat.
Likelihood vs Probability:
While probability f(x | theta) views theta as fixed and evaluates the probability of outcomes x, the likelihood function L(theta | x) views observed data x as fixed and evaluates the plausibility of parameter values theta.
Log-Likelihood Simplification:
Because the natural logarithm is a strictly monotonically increasing function, maximizing L(theta) is mathematically identical to maximizing the log-likelihood l(theta) = ln L(theta):
l(theta) = Sum_{i=1}^n ln f(x_i; theta).
Converting products into sums drastically simplifies differential calculus.
2. Worked Practice Problem: Gaussian MLE Derivation
Notice that the MLE estimator for variance sigma^2 divides by n, not n-1.
Problem: For an i.i.d. sample x1, x2, ..., xn drawn from N(mu, sigma^2), derive the Maximum Likelihood Estimator for the mean parameter mu.
Step 1: Write the Gaussian likelihood function:
L(mu) = Product_{i=1}^n [ 1 / (sigma sqrt(2pi)) ] exp[ - (x_i - mu)^2 / (2 sigma^2) ]
L(mu) = (2 pi sigma^2)^{-n/2} exp[ - (1 / (2 sigma^2)) * Sum_{i=1}^n (x_i - mu)^2 ]
Step 2: Take the natural logarithm to form the log-likelihood l(mu):
l(mu) = - (n / 2) ln(2 pi sigma^2) - [ 1 / (2 sigma^2) ] * Sum_{i=1}^n (x_i - mu)^2
Step 3: Differentiate with respect to mu and set to zero:
d l(mu) / d mu = - [ 1 / (2 sigma^2) ] Sum_{i=1}^n [ 2 (x_i - mu) (-1) ] = 0
= [ 1 / sigma^2 ] * Sum_{i=1}^n (x_i - mu) = 0
Step 4: Solve for mu:
Sum_{i=1}^n x_i - n mu = 0 => n mu = Sum_{i=1}^n x_i
mu_hat_{MLE} = (1 / n) * Sum_{i=1}^n x_i = X_bar
Conclusion: The Maximum Likelihood Estimator for the Gaussian mean is the sample mean X_bar.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.