Context Windows & Long Context
What is the Context Window?
The context window is the maximum number of tokens an LLM can process in a single forward pass — both input and output combined. Modern context windows: • GPT-4o: 128K tokens • Claude Opus 4.8: 1M tokens • Gemini 1.5 Pro: 2M tokens 1 token ≈ 4 characters ≈ 0.75 words in English. 100K tokens ≈ a full novel.
Lost in the Middle Problem
Research shows LLMs perform best on content at the beginning and end of the context. Information buried in the middle of a long context is more likely to be ignored. Mitigation strategies: • Place the most important instructions at the start AND end • Use RAG to inject only the most relevant chunks rather than full documents • Summarize and compress earlier parts of long conversations
Handling Long Documents
Map-Reduce: Process each chunk independently (Map), then combine results (Reduce). Good for summarization. Refine: Process first chunk, then iteratively refine the result with each subsequent chunk. Better coherence. Hierarchical: Summarize chunks into higher-level summaries, then summarize summaries.
For RAG, chunk size of 256–512 tokens with 10-20% overlap works well for most document types. Use semantic chunking for better results.
Finished reading? Mark it complete to earn your XP.