RNNs, LSTMs & GRUs
Recurrent Neural Networks
RNNs process sequential data by maintaining a hidden state h_t across time steps: h_t = tanh(W_h · h_{t-1} + W_x · x_t + b) The same weights are used at every timestep. Output can be: • Many-to-one: sentiment analysis (all inputs → one label) • One-to-many: image captioning (one image → sequence of words) • Many-to-many: translation, POS tagging
Vanishing Gradient Problem
During backpropagation through time (BPTT), gradients are multiplied by W_h at every step. If |W_h| < 1, gradients shrink exponentially. If |W_h| > 1, they explode. Result: Vanilla RNNs can only remember ~10 steps back. They fail at long-range dependencies.
LSTM — Long Short-Term Memory
LSTM introduces a cell state C_t (long-term memory) with gating mechanisms: Forget gate: f_t = σ(W_f · [h_{t-1}, x_t] + b_f) — what to erase Input gate: i_t = σ(W_i · [h_{t-1}, x_t] + b_i) — what to write Cell update: C̃_t = tanh(W_C · [h_{t-1}, x_t] + b_C) New cell: C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t Output gate: o_t = σ(W_o · [h_{t-1}, x_t] + b_o) Hidden state: h_t = o_t ⊙ tanh(C_t)
GRU — Gated Recurrent Unit
GRU is a simplified LSTM with only 2 gates (reset and update). Fewer parameters, trains faster, comparable performance to LSTM on most tasks.
Sequence Modeling with LSTMs
For NLP tasks today, Transformers outperform LSTMs. But LSTMs still excel at streaming time-series, sensor data, and tasks needing strict causality.
Finished reading? Mark it complete to earn your XP.