Batch Normalization & Dropout
Batch Normalization
BatchNorm normalizes layer inputs across the batch dimension: x̂ = (x − μ_B) / √(σ²_B + ε) y = γ·x̂ + β μ_B, σ²_B: batch mean and variance (computed during training) γ, β: learnable scale and shift Benefits: • Reduces internal covariate shift — makes training stable • Allows much higher learning rates • Acts as mild regularizer • Reduces sensitivity to weight initialization
Layer Norm vs Batch Norm
BatchNorm: normalizes over the batch dimension. Great for CNNs. Bad for variable-length sequences or small batches. LayerNorm: normalizes over the feature dimension per sample. Standard in Transformers. Works with batch_size=1. GroupNorm: normalizes over groups of channels. Good for object detection with small batch sizes. InstanceNorm: normalizes per sample per channel. Used in style transfer.
Dropout
Dropout randomly zeros neuron outputs with probability p during training. Forces the network to learn redundant representations — acts as an ensemble of 2^N sub-networks. Training: scale surviving activations by 1/(1-p) Inference: no dropout, all neurons active Typical rates: p=0.1–0.3 for hidden layers, p=0.5 for large FC layers. Dropout is less effective in CNNs — use SpatialDropout2D or BatchNorm instead.
BatchNorm & Dropout in Practice
Always call model.eval() before inference and model.train() before training. Forgetting this is one of the most common bugs in PyTorch.
Finished reading? Mark it complete to earn your XP.