Contents
Figure 1: Decoding Strategies Explained: Greedy, Beam Search, Top-k, Top-p, and Temperature — the model produces a distribution, and the decoder decides which slice of it you ever see
Ask a language model the same question twice, and you might get two different answers. Turn one setting, and it repeats itself like a broken record. Turn another, and it writes poetry.
Those settings do not change the model at all. They change how you pick each next token from the model's output. Here is every major decoding strategy, with a toy example you can run, and the settings to reach for.
The Setup: Logits, Probabilities, and a Choice
At each step, a language model scores every token in its vocabulary. Those scores (logits) become a probability distribution through softmax. Then a separate piece of code — the decoding strategy — chooses which token to emit. Greedy, beam search, and sampling are three versions of that code, and temperature and top-p act as dials on it.
Strategies fall into two families:
- Deterministic: greedy and beam search, which give reproducible output.
- Stochastic: top-k, top-p, and temperature, which give varied output.
Here is the toy distribution used throughout. Eight tokens, made-up logits:
import numpy as np
tokens = np.array(["the", "a", "cat", "dog", "pizza", "quantum", "xylophone", "zzz"])
logits = np.array([3.0, 2.5, 2.0, 1.0, 0.2, -0.5, -1.5, -3.0])
def softmax(z):
z = z - z.max()
e = np.exp(z)
return e / e.sum()
p = softmax(logits)
The measured probabilities:
the 0.4516 a 0.2739 cat 0.1661 dog 0.0611
pizza 0.0275 quantum 0.0136 xylophone 0.0050 zzz 0.0011
Every number below comes from running this code, not from memory.
Greedy Decoding
Greedy decoding picks the single most probable token at every step. In the toy example it always chooses the:
greedy_token = tokens[np.argmax(p)] # "the"
It is fast and reproducible, and it gives zero diversity. It also falls into repetition easily. The Hugging Face generation guide calls repetition a very common problem, and worse in greedy and beam search than in sampling.
Greedy has a second, subtler flaw: it only looks one step ahead. The best token now may lead to a poor sentence later. That is what beam search tries to fix.
Beam Search
Beam search keeps several candidate sequences alive at once. At each step it extends every candidate, scores the results, and keeps only the w best, where w is the beam width. Typical widths run 4 to 8, and cost grows roughly in proportion to the width.
Here is a tiny language model where greedy gets trapped. After the start token, A looks better than B (0.6 vs 0.4), but B leads to a much stronger continuation:
lm = {
"<s>": {"A": 0.6, "B": 0.4},
"A": {"C": 0.4, "D": 0.3, "E": 0.3},
"B": {"F": 0.9, "G": 0.1},
}
Running both strategies gives:
greedy / beam=1: <s> A C probability 0.24
beam=2: <s> B F probability 0.36
Greedy commits to A and ends up with a sequence of probability 0.24. Beam search with width 2 keeps B alive and finds the better path at 0.36. Beam search works especially well for structured tasks like machine translation, where one best answer exists.
But there is a catch for open-ended writing. Holtzman and colleagues found that decoding strategies optimising for high probability, like beam search, produce degenerate text even with a strong model like GPT-2 Large [2]. The most probable sequence often reads as bland and repetitive. Human text is not the most probable text — it is surprising in places, which is why chat models default to sampling.
Temperature
Temperature reshapes the distribution before you pick. It divides the logits by T, then applies softmax. Higher temperatures flatten the distribution; lower temperatures sharpen it toward the most probable tokens.
def temperature_probs(logits, T):
return softmax(logits / T)
On the toy logits, the top token's probability moves like this:
T=0.5: the 0.66 a 0.24 cat 0.09
T=1.0: the 0.45 a 0.27 cat 0.17 dog 0.06
T=2.0: the 0.30 a 0.23 cat 0.18 dog 0.11 pizza 0.07
As T approaches 0, the model behaves like greedy decoding. As T grows very large, the distribution approaches uniform, so every token becomes equally likely and the output turns to noise. Temperature is the most intuitive dial: low for precision, high for creativity.
Temperature has a limit, though. It reshapes the whole distribution but does not cut off the unreliable tail. Even at moderate temperatures, rare junk tokens keep a small chance of appearing, and across thousands of tokens one of them eventually gets picked.
Top-k Sampling
Top-k tackles the tail directly. It keeps only the k most probable tokens, renormalises, and samples from them. Fan et al. proposed it in 2018 [1]. Truncating the distribution reduces the risk of sampling from the long tail of low-probability, incoherent tokens while keeping some diversity.
def top_k_probs(p, k):
out = np.zeros_like(p)
idx = np.argsort(p)[::-1][:k]
out[idx] = p[idx]
return out / out.sum()
With k=3, the toy distribution becomes the 0.51, a 0.31, cat 0.19. Set k=1 and you are back to greedy decoding, since a shortlist of one leaves no choice.
The weakness is that k stays fixed. When the model is very sure, a fixed k of 50 still lets in 49 bad options. When the model is genuinely unsure, a fixed k of 3 cuts off good ones.
Top-p (Nucleus) Sampling
Top-p fixes that by letting the shortlist size adapt. Holtzman et al. introduced nucleus sampling in 2019 [2]. It samples from the smallest set of tokens whose cumulative probability exceeds a threshold p.
def top_p_probs(p, top_p):
order = np.argsort(p)[::-1]
cum = np.cumsum(p[order])
cutoff = np.searchsorted(cum, top_p) + 1
keep = order[:cutoff]
out = np.zeros_like(p)
out[keep] = p[keep]
return out / out.sum()
On the toy distribution, top_p=0.9 keeps four tokens: the 0.47, a 0.29, cat 0.17, dog 0.06.
Now the adaptive behaviour. I built one peaked distribution and one flat one:
nucleus size with p=0.9: peaked → 1 token flat → 8 tokens
When the model is confident, the nucleus shrinks to a single token. When it is uncertain, the nucleus widens — all the way to the full vocabulary here, because seven of eight uniform tokens only reach 0.875, which is below the 0.9 threshold.
That adaptivity is why top-p usually beats a fixed top-k. Holtzman's group also showed that high values of k are needed to match human text statistics, which causes incoherence, while nucleus sampling matches reference perplexity by tuning p without that problem [2].
Why Maximising Probability Backfires
Here is the surprising lesson from the research. Maximisation-based decoding like beam search tends to produce degenerate outputs with repetition and blandness, while pure sampling often yields incoherence. Text from maximisation or top-k sampling can come out too probable, which signals a lack of diversity compared with human writing.
So good open-ended generation sits in a middle zone: random enough to stay human-like, constrained enough to avoid garbage. Top-p and temperature together hit that zone for most use cases. Researchers still describe the best decoding strategy as an open question, so do not expect a universal winner.
Newer Methods Worth Knowing
- Min-p sampling: sets a threshold relative to the top token's probability, so a token must reach some fraction (say 5%) of the best token's probability to stay. It adapts harder than top-p when the model is confident.
- Mirostat: adapts the shortlist dynamically to hold a target perplexity.
- Contrastive decoding: uses the probability gap between a large and a small model to improve quality.
- Repetition penalties: shrink the probabilities of tokens already generated. Values around 1.1 to 1.2 are common, and large values produce unnatural word choices.
Speculative decoding also deserves a mention, but it speeds up generation without changing the output distribution, so it solves a different problem entirely.
Settings Cheat Sheet
These starting points are reasonable defaults, so tune them for your model:
| Use case | Setting |
|---|---|
| Code, math, factual extraction | temperature 0 to 0.2, or greedy |
| General chat | temperature ~0.7, top-p ~0.9 |
| Creative writing | temperature 1.0+, top-p ~0.95 |
| Translation or speech | beam search, width 4–8 |
| Reproducible tests | greedy, plus a pinned model version |
One practical note: some APIs recommend changing temperature or top-p, but not both at once. APIs also differ in which knobs they expose, so check your provider's documentation rather than assuming.
Using These in Hugging Face
The Transformers library exposes all of these through generate:
greedy = model.generate(**inputs, do_sample=False, max_new_tokens=40)
beam = model.generate(**inputs, num_beams=5, max_new_tokens=40)
nucleus = model.generate(
**inputs,
do_sample=True,
top_p=0.9,
top_k=0, # disable top-k so only top-p applies
temperature=0.8,
max_new_tokens=40,
)
Note that do_sample=True is what activates top-p and temperature at all — with sampling off, those arguments are ignored. top_k=0 disables top-k, which is the library's convention.
Try the same prompt with all three and compare. The greedy and beam outputs often look repetitive on open-ended prompts, which makes the problem easy to see in about thirty seconds.
Common Pitfalls
- Setting temperature to 0 and expecting creativity. You get greedy decoding.
- Using beam search for chat or stories. Outputs turn bland and repetitive.
- Stacking every knob at once. Several truncation methods interact in ways that are hard to debug.
- Cranking temperature to fix repetition. A mild repetition penalty or top-p often works better.
- Assuming deterministic means correct. Greedy output can still be wrong, just consistently wrong.
- Ignoring the order of operations. Libraries differ on whether temperature is applied before or after truncation.
- Forgetting
do_sample=True. Without it,top_pandtemperaturedo nothing.
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Natural Language Processing with Transformers | the generation chapter walks through greedy, beam, and sampling with the exact Hugging Face arguments used above. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read it after this post if you want the library-level details: how generate composes logits processors, why the order of operations matters, and how to build a custom decoder. It is the practical companion to the theory here.
Wrapping Up
Greedy takes the top token, beam search tracks several sequences, temperature reshapes the distribution, top-k keeps a fixed shortlist, and top-p keeps an adaptive one. For open-ended text, sample with moderate temperature and top-p. For structured tasks with one right answer, go deterministic.
Take the toy code above, change the logits, and watch the nucleus grow and shrink. Then run a real model with three settings on the same prompt.
Which setting will you try first?
Frequently Asked Questions
What is the difference between top-k and top-p sampling?
Top-k keeps a fixed number of candidates, so a shortlist of 50 always has 50 entries whether the model is confident or confused. Top-p keeps the smallest set whose cumulative probability passes a threshold, so the shortlist shrinks when the model is sure and widens when it is not.
Does temperature change the model?
No. Temperature only divides the logits before softmax, which reshapes the distribution the model already produced. The weights, the context, and the computation are identical; only the sampling distribution changes.
Why is beam search bad for chatbots?
Beam search maximises sequence probability, and the most probable sequence tends to be bland, repetitive, and short on surprise. Holtzman et al. showed that likelihood-maximising decoding produces degenerate text even from a strong model, which is why chat systems sample instead.
Should I set temperature to 0 for deterministic output?
That works, but greedy decoding is the cleaner way to say it. Some APIs treat temperature 0 as a special case, and a few reject it. If you need reproducibility, use greedy decoding and pin the model version, since providers update weights.
Can I use temperature and top-p together?
Yes, and most chat defaults do. Temperature reshapes the distribution and top-p truncates the tail; applying temperature first then top-p is the common order. Avoid stacking top-k, top-p, and a repetition penalty at once, because the interactions become hard to reason about.
Sources
[1]Fan, Lewis, and Dauphin, Hierarchical Neural Story Generation — the paper that introduced top-k sampling: arxiv.org[2]Holtzman et al., The Curious Case of Neural Text Degeneration — nucleus sampling and the case against maximisation: arxiv.org[3]Hugging Face, How to generate text: decoding methods — greedy, beam, and sampling with the library arguments: huggingface.co[4]Hugging Face, Generation strategies — the currentgeneratedocumentation and logits processors: huggingface.co[5]Tunstall, von Werra, and Wolf, Natural Language Processing with Transformers — the generation chapter in book form: amazon.com
