Activation Functions

Without activation functions, N stacked linear layers collapse to a single linear transformation. Non-linearity lets networks approximate any continuous function (Universal Approximation Theorem).

ReLU: f(x) = max(0, x) + Computationally cheap, no vanishing gradient for x>0 − Dead neurons: if x<0 always, the neuron never activates or updates Leaky ReLU: f(x) = max(0.01x, x) — fixes dead neurons PReLU: learnable slope for negative values ELU: smooth negative saturation, self-normalizing properties SELU: scaled ELU, enables self-normalizing networks (no BatchNorm needed)

GELU (Gaussian Error Linear Unit): f(x) = x · Φ(x) — used in BERT, GPT, ViT SwiGLU: f(x, gate) = x · sigmoid(gate) — used in LLaMA, PaLM Mish: f(x) = x · tanh(softplus(x)) — smooth, outperforms ReLU on many tasks