Neural Networks Fundamentals

A Neural Network is a universal function approximator built from layers of interconnected neurons. Each neuron computes: output = activation(W · x + b) Where W is a weight matrix, x is the input, b is a bias, and activation is a non-linear function. Stacking many layers allows the network to learn arbitrarily complex mappings from input to output.

Input Layer — Receives raw features. One neuron per feature. Hidden Layers — Extract progressively abstract representations. More layers = deeper network = more expressive. Output Layer — Produces predictions. Shape depends on task: • Binary classification: 1 neuron + Sigmoid • Multi-class: N neurons + Softmax • Regression: 1 neuron + no activation (linear)

Without activation functions, stacking layers is equivalent to a single linear transformation. Non-linearities allow the network to learn curves, boundaries, and complex patterns. ReLU: f(x) = max(0, x) — default for hidden layers. Fast, no vanishing gradient for positive values. Leaky ReLU: f(x) = max(0.01x, x) — avoids dead neurons. Sigmoid: f(x) = 1/(1+e^−x) — used in output for binary classification. Softmax: normalizes outputs to probability distribution for multi-class. GELU: Gaussian Error Linear Unit — used in Transformers (BERT, GPT).