Bias-Variance Tradeoff

The expected prediction error decomposes into three terms: Error = Bias² + Variance + Irreducible Noise Bias — Error from wrong assumptions (model too simple). Underfitting. Variance — Error from sensitivity to training data fluctuations (model too complex). Overfitting. Irreducible Noise — Inherent data noise, cannot be reduced by any model.

High-bias models make strong assumptions and miss the true pattern. Signs: Training error is high. Test error ≈ training error. Causes: Model too simple (e.g., linear model on non-linear data), too few features. Fix: More complex model, more features, reduce regularization.

High-variance models memorize training data including noise. Signs: Training error very low, test error much higher. Causes: Model too complex (deep tree, large neural network), too little data. Fix: More data, regularization (L1/L2/dropout), simpler model, early stopping, cross-validation.