The Law of Total Probability & Bayes' Theorem
“Partitions of sample space, marginalization, prior and posterior probabilities, and likelihood ratios.”
1.Inverse Probability & The Bayesian Mechanism
§ 1.1.In machine learning, we often observe evidence (data D) and wish to infer the unobserved generative hypothesis or class label C.
The Law of Total Probability:
Let events A_1, A_2, ..., A_k form a partition of the sample space Omega (meaning they are mutually disjoint A_i ∩ A_j = empty and collectively exhaustive Union A_i = Omega). Then for any event B:
P(B) = Sum_{i=1}^k P(B ∩ A_i) = Sum_{i=1}^k P(B | A_i) P(A_i).
Bayes' Theorem Formulation:
Substituting the total probability of B into the denominator of the conditional definition P(A_k | B) = P(A_k ∩ B) / P(B) yields:
P(A_k | B) = [ P(B | A_k) P(A_k) ] / [ Sum_{i=1}^k P(B | A_i) P(A_i) ].
Here, P(A_k) is the Prior, P(B | A_k) is the Likelihood, P(B) is the Evidence / Marginal Likelihood, and P(A_k | B) is the Posterior.
2. Worked Practice Problem: Rare Disease Diagnostic & Base Rate Fallacy
Do not confuse P(Positive | Disease) = 0.98 with the desired posterior P(Disease | Positive).
Problem: A rare medical disease D affects 0.5% of a population (prior P(D) = 0.005). A diagnostic test has 98% sensitivity (P(+ | D) = 0.98) and 95% specificity (P(- | D^c) = 0.95). If a randomly chosen patient tests positive, what is the posterior probability that they actually have the disease?
Step 1: Define given probabilities and complements:
P(D) = 0.005 => P(D^c) = 1 - 0.005 = 0.995
P(+ | D) = 0.98 (True Positive Rate / Sensitivity)
P(- | D^c) = 0.95 (Specificity) => P(+ | D^c) = 1 - 0.95 = 0.05 (False Positive Rate)
Step 2: Compute total probability of a positive test P(+):
P(+) = P(+ | D) P(D) + P(+ | D^c) P(D^c)
P(+) = (0.98 0.005) + (0.05 0.995)
P(+) = 0.0049 + 0.04975 = 0.05465
Step 3: Apply Bayes' Theorem:
P(D | +) = [ P(+ | D) * P(D) ] / P(+)
P(D | +) = 0.0049 / 0.05465 = 490 / 5465 ≈ 0.08966 (8.97%).
Conclusion: Despite a 98% sensitive and 95% specific test, a positive test result yields only an ~8.97% chance of having the disease because the prior base rate (0.5%) is so small.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.