Ensemble Learning: Random Forests, Bagging & AdaBoost
“Bootstrap aggregation, decorrelating trees, out-of-bag validation, and adaptive sample reweighting.”
1.The Variance-Reduction Mechanics of Ensembles
§ 3.1.Bagging (Bootstrap Aggregation):
Draws B independent bootstrap samples with replacement from training dataset D, fits a deep high-variance tree to each, and aggregates by voting (classification) or averaging (regression).
If individual trees have variance sigma^2 and pairwise correlation rho, the ensemble variance is: Var(Ensemble) = rho sigma^2 + [ (1 - rho) / B ] sigma^2.
Random Forest Innovation: By forcing each tree node to choose from a random subset of sqrt(d) features, trees are decorrelated, significantly lowering rho and crushing variance.
2. Worked Practice Problem: AdaBoost Classifier Weighting
If an error rate eps_m >= 0.5, the weak learner performs no better than random guessing.
Problem: An AdaBoost weak learner produces a training error rate eps = 0.20 on weighted training data. Compute the stage weight alpha assigned to this learner.
Step 1: Write AdaBoost weight formula:
alpha = (1/2) * ln[ (1 - eps) / eps ]
Step 2: Substitute eps = 0.20:
(1 - 0.20) / 0.20 = 0.80 / 0.20 = 4.0
Step 3: Compute natural logarithm:
ln(4.0) ≈ 1.3863
Step 4: Compute alpha:
alpha = (1/2) * 1.3863 = 0.69315.
Interpretation: Because the learner error was low (20%), it receives a positive vote weight of ~0.69 in the final majority decision.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.