Logistic Regression, Logit Link & Cross-Entropy Optimization
“Sigmoid mapping, maximum likelihood estimation, odds ratios, and linear decision boundaries.”
1.The Probabilistic Derivation of Logistic Regression
§ 2.1.Unlike linear regression, which predicts an unbounded real value y in R, binary classification requires mapping predictions to the probability interval [0, 1].
Maximum Likelihood Estimation (MLE):
Assuming training examples are independently Bernoulli-distributed:
L(w) = Product_{i=1}^n p_i^{y_i} (1 - p_i)^{1 - y_i}.
Taking the natural log gives the Log-Likelihood:
ln L(w) = Sum_{i=1}^n [ y_i ln(p_i) + (1 - y_i) ln(1 - p_i) ].
Minimizing the negative log-likelihood per sample yields the Binary Cross-Entropy Loss function.
2. Worked Practice Problem: Odds and Probability Calculation
Do not confuse Odds = p / (1-p) with Probability p = Odds / (1 + Odds).
Problem: A logistic regression model outputs w^T x = 1.386 for an input instance. What are the predicted odds and probability p that the instance belongs to the positive class?
Step 1: Compute Odds:
Odds = e^{w^T x} = e^{1.386} ≈ 4.0
Step 2: Compute Probability p:
p = Odds / (1 + Odds) = 4 / (1 + 4) = 4 / 5 = 0.80 (80%).
Verification via Sigmoid:
sigma(1.386) = 1 / (1 + e^{-1.386}) = 1 / (1 + 0.25) = 1 / 1.25 = 0.80.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.