BERT vs GPT — Encoder vs Decoder
BERT (Bidirectional Encoder Representations from Transformers)
Google, 2018. Encoder-only Transformer trained on: 1. Masked Language Modeling (MLM): Randomly mask 15% of tokens, predict the masked tokens. Forces bidirectional context understanding. 2. Next Sentence Prediction (NSP): Predict if sentence B follows sentence A. BERT reads the entire sequence at once — each token attends to all others (bidirectional). Output: rich contextual embeddings for every token.
GPT (Generative Pre-trained Transformer)
OpenAI, 2018. Decoder-only Transformer trained with Causal Language Modeling: Predict next token given all previous tokens: P(x_t | x_1,...,x_{t-1}) Causal mask prevents attending to future tokens. Output: next token probability distribution.
When to Use Which
Use BERT (encoder) for: • Text classification (sentiment, spam) • Named Entity Recognition (NER) • Question Answering (extractive) • Semantic similarity, embeddings • Token-level tasks Use GPT (decoder) for: • Text generation, summarization • Code generation • Chatbots and conversational AI • Few-shot prompting • Completion tasks
Fine-tuning BERT for Classification
Fine-tuning LR should be 1e-5 to 5e-5. Higher LR catastrophically forgets pretrained knowledge (catastrophic forgetting).
Finished reading? Mark it complete to earn your XP.