Applied AI·Local and self-hosted inference
a small model guesses the next few tokens, the big one checks them in a single pass, and you get identical output faster.
Speculative decoding
Draft summary, pending review
A small draft model proposes several tokens; the large model verifies the batch in one pass, keeping the correct prefix. Output is identical to the large model alone, just faster. A useful reminder that inference speed is an engineering surface, not a fixed property.