Covariance, Correlation & Bivariate Gaussian Properties
“Covariance Cov(X, Y), Pearson correlation rho, Positive Semi-Definite covariance matrices, and bivariate Gaussian independence.”
1.Linear Dependence: Covariance & Correlation
§ 3.1.Covariance measures the joint linear variability of two random variables.
Properties of Covariance:
1. Cov(X, Y) = E[XY] - E[X] E[Y].
2. If X and Y are independent: E[XY] = E[X] E[Y] => Cov(X, Y) = 0.
Pearson Correlation Coefficient rho:
Normalized scale-invariant measure of linear association:
rho = Cov(X, Y) / (sigma_X sigma_Y).
By Cauchy-Schwarz inequality: -1 <= rho <= +1.
- rho = +1: Perfect positive linear relationship (Y = aX + b, a > 0).
- rho = -1: Perfect negative linear relationship (Y = aX + b, a < 0).
- rho = 0: Uncorrelated (no linear relationship).
The Bivariate Gaussian Special Case:
For jointly Gaussian random variables, uncorrelatedness strictly implies independence, because cross-terms in the exponent of the bivariate normal PDF vanish when rho = 0.
2. Worked Practice Problem: Variance of Linear Combination
Notice the negative coefficient on Y: (-3)^2 = +9, and the cross-term has 2 * (2) * (-3) * Cov(X, Y).
Problem: Random variables X and Y have Var(X) = 16, Var(Y) = 25, and Pearson correlation coefficient rho = 0.6. Find the variance of the linear combination W = 2X - 3Y + 7.
Step 1: Compute standard deviations of X and Y:
sigma_X = sqrt(16) = 4
sigma_Y = sqrt(25) = 5
Step 2: Compute Cov(X, Y):
Cov(X, Y) = rho sigma_X sigma_Y = 0.6 4 5 = 0.6 * 20 = 12.
Step 3: State the variance expansion for Var(aX + bY + c):
Constants do not affect variance: Var(W) = Var(2X - 3Y)
Var(aX + bY) = a^2 Var(X) + b^2 Var(Y) + 2 a b * Cov(X, Y)
Step 4: Substitute coefficients a = 2, b = -3:
Var(W) = (2)^2 Var(X) + (-3)^2 Var(Y) + 2 (2) (-3) * Cov(X, Y)
Var(W) = 4 (16) + 9 (25) - 12 * Cov(X, Y)
Var(W) = 64 + 225 - 12 * (12) = 289 - 144 = 145.
Conclusion: The variance of 2X - 3Y + 7 is exactly 145.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.