Temperature, Top-k, Top-p Sampling
How LLMs Generate Text
At each step, the LLM outputs a probability distribution over the entire vocabulary (50k+ tokens). Sampling strategy determines how we pick the next token from that distribution. Greedy decoding: always pick the highest probability token. Fast but repetitive and often suboptimal. Beam search: keep top-k sequences at each step. Better quality but computationally expensive.
Temperature
Temperature τ scales the logits before softmax: P(token_i) = exp(logit_i / τ) / Σ exp(logit_j / τ) τ = 1.0: standard distribution (default) τ < 1.0: sharper distribution — more deterministic, focused, conservative τ > 1.0: flatter distribution — more random, creative, diverse τ → 0: equivalent to greedy decoding τ → ∞: uniform distribution (completely random) Use low temperature (0.0–0.3) for factual tasks. Higher (0.7–1.0) for creative tasks.
Top-k Sampling
Keep only the top-k most probable tokens, redistribute probability mass among them, then sample. k=1: greedy decoding k=50: sample from top 50 tokens only Problem: k is fixed regardless of how peaked or flat the distribution is.
Top-p (Nucleus) Sampling
Keep the smallest set of tokens whose cumulative probability ≥ p, then sample. p=0.9: include tokens until their combined probability reaches 90% Adaptive: if the distribution is peaked (model is confident), fewer tokens are included. If flat (uncertain), more tokens are included. Superior to top-k in practice.
Sampling in Code
For production: temperature=0.7, top_p=0.9 is a solid starting point. For code generation: temperature=0.2. For creative writing: temperature=1.0.
Finished reading? Mark it complete to earn your XP.