Model Evaluation: Confusion Matrix, ROC-AUC & Bias-Variance Decomposition
“Precision, Recall, Specificity, F1 harmonic mean, ROC curves, and the mathematical bias-variance decomposition.”
1.Confusion Matrix Metrics & ROC Mechanics
§ 6.1.Evaluation metrics determine whether a learned machine learning hypothesis generalizes to unseen test distributions.
The Four Fundamental Quadrants:
- True Positive (TP): Ground truth 1, Predicted 1.
- False Positive (FP, Type I Error): Ground truth 0, Predicted 1.
- True Negative (TN): Ground truth 0, Predicted 0.
- False Negative (FN, Type II Error): Ground truth 1, Predicted 0.
The Bias-Variance Tradeoff:
- Bias: Error introduced by approximating complex real-world phenomena with overly simplistic models (underfitting).
- Variance: Model sensitivity to random fluctuations in the training set (overfitting).
2. Worked Practice Problem: Complete Evaluation Metric Derivation
Recall has False Negatives (FN) in the denominator; Precision has False Positives (FP) in the denominator.
Problem: A diagnostic test evaluates 200 patients. Confusion matrix: TP = 70, FP = 10, FN = 30, TN = 90. Compute Precision, Recall, Specificity, and F1-Score.
Step 1: Compute Precision (P):
P = TP / (TP + FP) = 70 / (70 + 10) = 70 / 80 = 0.875 (87.5%).
Step 2: Compute Recall (R, Sensitivity, TPR):
R = TP / (TP + FN) = 70 / (70 + 30) = 70 / 100 = 0.700 (70.0%).
Step 3: Compute Specificity (TNR):
Specificity = TN / (TN + FP) = 90 / (90 + 10) = 90 / 100 = 0.900 (90.0%).
False Positive Rate (FPR) = 1 - Specificity = 1 - 0.90 = 0.100 (10.0%).
Step 4: Compute F1-Score:
F1 = 2 (P R) / (P + R) = 2 (0.875 0.70) / (0.875 + 0.70) = 2 * (0.6125) / (1.575) = 1.225 / 1.575 = 7 / 9 ≈ 0.7778 (77.78%).
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.