Backpropagation

Backpropagation is the algorithm that computes gradients of the loss with respect to every weight in a neural network using the chain rule of calculus. It enables gradient descent to update all weights efficiently in a single backward pass.

Input flows through the network layer by layer: x → z₁ = W₁x + b₁ → a₁ = relu(z₁) → z₂ = W₂a₁ + b₂ → ŷ = sigmoid(z₂) → L = loss(ŷ, y) All intermediate values (z₁, a₁, z₂) are cached — needed for the backward pass.

Starting from the loss, propagate gradients backward: ∂L/∂W₂ = ∂L/∂ŷ · ∂ŷ/∂z₂ · ∂z₂/∂W₂ ∂L/∂a₁ = ∂L/∂ŷ · ∂ŷ/∂z₂ · ∂z₂/∂a₁ ∂L/∂W₁ = ∂L/∂a₁ · ∂a₁/∂z₁ · ∂z₁/∂W₁ Each weight gets its exact gradient. Then update: W ← W − α·∂L/∂W