Optimizers — SGD, Adam, AdamW

Vanilla SGD oscillates. Momentum smooths updates by accumulating a velocity: v_t = β·v_{t-1} + (1−β)·∇L w_t = w_{t-1} − α·v_t β=0.9 is standard. Nesterov Momentum looks ahead before computing gradient — faster convergence.

Adam maintains per-parameter adaptive learning rates: m_t = β₁·m_{t-1} + (1−β₁)·g (1st moment — momentum) v_t = β₂·v_{t-1} + (1−β₂)·g² (2nd moment — squared gradient) ŵ_t = w_{t-1} − α·m̂_t / (√v̂_t + ε) Defaults: α=1e-3, β₁=0.9, β₂=0.999, ε=1e-8 Adam handles sparse gradients, scales LR per parameter, converges fast. Default choice for most deep learning.

Standard Adam applies weight decay incorrectly (via gradient update). AdamW decouples weight decay from gradient: w_t = (1 − α·λ)·w_{t-1} − α·m̂_t / (√v̂_t + ε) This is the correct L2 regularization for Adam. Used in all modern Transformers (BERT, GPT, ViT). Always prefer AdamW over Adam for transformer training.