Gradient Descent

Gradient Descent is the core optimization algorithm for training ML models. It minimizes a loss function L(w) by iteratively adjusting weights in the direction of steepest descent (negative gradient). Update rule: w ← w − α · ∇L(w) Where α is the learning rate and ∇L(w) is the gradient of the loss.

Batch GD — Computes gradient on the full dataset. Accurate but extremely slow for large datasets. Stochastic GD (SGD) — Computes gradient on one sample at a time. Fast updates but very noisy — oscillates around the minimum. Mini-Batch GD — Uses batches of 32–512 samples. Best of both: stable convergence, GPU-friendly. Used in all deep learning frameworks.

Too high → loss explodes or oscillates, never converges. Too low → converges extremely slowly, may get stuck in local minima. Solutions: Learning rate schedules (step decay, cosine annealing), adaptive optimizers (Adam, RMSprop).

Momentum accelerates SGD by accumulating a velocity vector: v ← βv − α∇L; w ← w + v Adam (Adaptive Moment Estimation) combines momentum + adaptive learning rates per parameter. It's the default optimizer for most deep learning tasks.