Multilayer Perceptrons, Non-Linear Activations & Vanishing Gradients
“Overcoming linear separability, Universal Approximation Theorem, and ReLU non-saturating gradients.”
1.The Architectural Foundations of Deep Learning
§ 5.1.In 1969, Minsky and Papert proved that single-layer perceptrons cannot learn simple non-linear logical functions such as XOR.
Why Multilayer Networks Succeed:
By composing linear affine transformations z = Wx + b with non-linear activation functions a = g(z), neural networks bend and fold feature space, allowing complex non-linear classification boundaries to become linearly separable at the final output layer.
2. Worked Practice Problem: XOR Architecture Minimum Parameters
Remember to count both weights and bias parameters in each layer.
Problem: What is the minimum number of hidden neurons in a 1-hidden layer MLP (with step or ReLU activation) required to solve the 2-input XOR problem?
Step 1: Understand XOR truth table:
Inputs: (0, 0)->0, (0, 1)->1, (1, 0)->1, (1, 1)->0.
Step 2: Linear boundaries required:
XOR requires TWO separating hyperplanes (e.g. x1 + x2 >= 0.5 AND x1 + x2 <= 1.5).
Step 3: Minimum Hidden Neurons = 2 hidden units.
Step 4: Count total trainable parameters (weights + biases):
Layer 1 (2 inputs -> 2 hidden): 2 x 2 weights + 2 biases = 6 parameters.
Layer 2 (2 hidden -> 1 output): 2 x 1 weights + 1 bias = 3 parameters.
Total Parameters = 6 + 3 = 9 parameters.
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.