Transformer Architecture

Introduced in 'Attention Is All You Need' (Vaswani et al., 2017). Replaced RNNs entirely for sequence modeling. Two key properties: 1. Parallelism — processes all tokens simultaneously (no sequential dependency) 2. Global receptive field — every token attends to every other token in O(1) steps

Each encoder layer: 1. Multi-Head Self-Attention(x, x, x) 2. Add & LayerNorm (residual connection) 3. Feed-Forward Network: FFN(x) = max(0, xW₁+b₁)W₂+b₂ 4. Add & LayerNorm FFN expands to 4× d_model then contracts. Acts as per-token processing. BERT uses 12 encoder layers.

Each decoder layer has 3 sub-layers: 1. Masked Multi-Head Self-Attention — can only attend to previous tokens (causal mask) 2. Cross-Attention — queries from decoder, keys/values from encoder output 3. Feed-Forward Network GPT uses decoder-only (no cross-attention). Used for generation.

Self-attention is permutation-invariant — it doesn't know token order. Positional encodings inject position information: PE(pos, 2i) = sin(pos / 10000^{2i/d_model}) PE(pos, 2i+1) = cos(pos / 10000^{2i/d_model}) Modern models use learned positional embeddings (BERT) or RoPE (Rotary Position Embedding, used in LLaMA, GPT-NeoX).