Comparison
Greedy decodingvsSampling
Greedy decoding
you always take the top token, get the same answer on every run, and occasionally watch it repeat itself in a loop.
Always picking the single most probable token. Maximally consistent, sometimes repetitive. Temperature 0 approximates this.
Full entry →Sampling
you draw the next token from the distribution instead of always taking the top one, which is why two identical calls disagree.
Choosing the next token from the probability distribution rather than always taking the top one. Sampling gives variety and creativity; it is also why the same prompt yields different answers on repeat runs.
Full entry →