Attention Mechanism

In seq2seq RNN models, the encoder compresses all input into a fixed-length vector — losing information for long sequences. Attention allows the decoder to 'look at' all encoder states and focus on the most relevant parts at each decoding step.

Three matrices computed from input X: • Query (Q) = X·W_Q — what are we looking for? • Key (K) = X·W_K — what does each position contain? • Value (V) = X·W_V — what to output from each position? Attention(Q,K,V) = softmax(Q·Kᵀ / √d_k) · V 1. Q·Kᵀ: similarity between every query and every key → (seq_len × seq_len) scores 2. /√d_k: scale to prevent softmax saturation in high dimensions 3. softmax: convert to probability distribution (attention weights) 4. ·V: weighted sum of values

Run H attention heads in parallel, each with different learned projections: MultiHead(Q,K,V) = Concat(head_1,...,head_H) · W_O head_i = Attention(Q·W_Qi, K·W_Ki, V·W_Vi) Each head learns to attend to different relationships (syntax, coreference, semantics, etc.).