The Backpropagation Algorithm: Multivariable Chain Rule Derivation
“Backward error propagation, intermediate sensitivity deltas, computational graphs, and SGD updates.”
1.The Step-by-Step Mathematical Derivation of Backprop
§ 5.1.Backpropagation is an efficient dynamic programming application of the multivariable calculus chain rule on a directed acyclic computational graph.
The Four Fundamental Backprop Equations:
1. delta^{(L)} = grad_a L * g'(z^{(L)}) (Output layer sensitivity).
2. delta^{(l)} = ( (W^{(l+1)})^T delta^{(l+1)} ) * g'(z^{(l)}) (Sensitivity backpropagated through layer weights).
3. dL / db^{(l)} = delta^{(l)} (Rate of change wrt bias).
4. dL / dW^{(l)} = delta^{(l)} (a^{(l-1)})^T (Outer product gradient wrt weight matrix).
2. Worked Practice Problem: 1-Step Backpropagation Weight Update
Evaluate the activation derivative g'(z) at the pre-activation value z.
Problem: A neuron has single input x = 2.0, weight w = 1.5, bias b = 0. Target output y = 1.0. Activation is linear g(z) = z, loss is squared error L = (1/2)(y_hat - y)^2. Learning rate eta = 0.1. Compute updated weight w_new.
Step 1: Forward Pass:
z = w x + b = 1.5 2.0 + 0 = 3.0
y_hat = g(z) = 3.0
Step 2: Compute Loss and Output Error:
Loss = (1/2) (3.0 - 1.0)^2 = (1/2) 4 = 2.0
dL / dy_hat = y_hat - y = 3.0 - 1.0 = 2.0
Step 3: Compute Gradient wrt Weight w:
dL / dw = (dL / dy_hat) (dy_hat / dz) (dz / dw)
dy_hat / dz = g'(z) = 1.0; dz / dw = x = 2.0
dL / dw = 2.0 1.0 2.0 = 4.0
Step 4: Update Weight via Gradient Descent:
w_new = w - eta (dL / dw) = 1.5 - 0.1 (4.0) = 1.5 - 0.4 = 1.1.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.