Speculative Decoding Explained: Draft, Verify and When It Stops Helping

The reply crawls out slowly, because a large model must run once for every single token it writes. A 300-token answer needs 300 runs, one after another. Speculative decoding is a fix for that wait, and vLLM ships it today.

Picture a common setup. A team runs an open-weight large model on one GPU with vLLM, an open-source serving engine, for a food-delivery app's support chat. The setup is made up, but the maths later in this guide is real.

At night only a few users chat at once. The GPU is far from busy, yet each reply still streams at about 25 tokens per second (made up). A 300-token reply takes 12 seconds, and users give up.

Speculative decoding lets the large model accept several tokens per run without changing what it would have written. You will see how it works, why it is safe, and when it quietly stops helping.

You need five words: token, forward pass, probability distribution, target model and draft model. Each one is short.

• Token: a small piece of text the model reads and writes, such as a word or part of a word. "Where is my order" is about four tokens.
• Forward pass: one run of the model over its input. Each pass gives a score for every possible next token.
• Probability distribution: the list of chances the model gives every possible next token. The chances add up to 1.
• Target model: the large, slow model whose answers you actually want. We call its distribution p.
• Draft model: a small, fast model that guesses what the target would say. We call its distribution q.

Sampling means picking the next token at random, weighted by those chances. Greedy decoding means always picking the single most likely token. Speculative decoding works for both.

Speculative decoding is a way to get several tokens out of one run of the large model. The small draft model guesses k tokens first. The large model then scores all of them in a single forward pass and keeps the ones it agrees with.

The idea appeared in two papers in late 2022 and early 2023. The first named the method speculative decoding; the second called it speculative sampling. Both describe the same draft-then-verify loop and the same accept rule.

One round, step by step

In our support chat, the user has typed "Where is my". The draft model guesses four tokens: "order", "right", "now", "please". Here k is 4, the number of guessed tokens per round.

• Draft: the small model runs four quick passes, one per guess, and remembers its chance q for each guessed token.
• Verify: the large model runs one pass over all four guesses, reusing its stored work for the earlier text. It returns its own chances p at all five positions.
• Accept or reject: going left to right, each guess is kept or rejected by the rule in the next section. The first rejection ends the round.
• Fix-up: at the rejected spot, the large model's chances give one replacement token. If all four were kept, the fifth position gives one bonus token for free.

The junior writer and the senior editor

Think of a junior writer and a senior editor. The junior quickly drafts the next four words of a reply. The editor reads all four at once and keeps them up to the first word she would not write.

At that word she writes her own choice and sends the rest back. The junior is the draft model, the editor is the target model, and reading four words at once is the single verify pass.

Where the picture breaks: a real editor judges words by taste. The target model uses a coin flip weighted by p and q, so even a good guess is sometimes rejected on purpose.

Because, at small batch sizes, a decoding step is limited by memory speed, not by maths. The GPU must read the model's weights from memory on every pass, whether it scores one position or five.

A batch is the group of user requests the server runs together in one pass. With one or a few requests, the maths per weight is tiny and the chip mostly waits for data. Adding four more positions adds maths, but barely adds memory reads.

The speculative sampling paper states this directly. It says sampling is typically memory-bandwidth bound, and that scoring K draft tokens in parallel has latency comparable to sampling one token. Its tests used K equal to 4 at batch size 1.

Keep this condition in mind. It is the whole reason the trick works, and it is the first thing that breaks when traffic grows.

It keeps a guessed token with probability min(1, p/q). If the guess is rejected, it picks a replacement from the leftover distribution max(0, p minus q), rescaled to add up to 1. Together these give exactly p.

The rule for one token

Look at the spot right after "Where is my". Our toy target and draft give these chances to four possible tokens. Real models choose from tens of thousands of tokens or more, but the rule is the same.

When the draft guesses "order", the large model likes it even more, so it is always kept. When the draft guesses "refund", it is over-confident: q is 0.40 but p is only 0.25. That guess survives only 62.5 percent of the time.

Here only "order" has leftover, so every replacement is "order". That makes sense. The draft under-predicts "order", and rejections hand that missing share back.

Why the total comes out as exactly p

Pause and predict: what is the chance that "order" ends up in the output? Try it before reading on.

There are two routes. The draft picks "order" with chance 0.30 and it is always kept, giving 0.30. A rejection happens with total chance 0.20, and every replacement is "order", adding 0.20. The total is 0.50, which is p.

For "refund", the draft picks it with chance 0.40 and keeps it 62.5 percent of the time. That gives 0.25, which is p again. The papers prove this for every token: kept share min(p, q) plus leftover share max(0, p minus q) equals p.

Two shortcuts look sensible but are wrong. Always keeping the guess gives you the draft model's answers, not the target's. Resampling from p itself after a rejection, instead of the leftover, double-counts tokens the draft already covered.

With greedy decoding the rule becomes simple. The target's p is 1 for its top token and 0 for the rest. So a guess is kept only if it equals the large model's top token. The output then matches plain greedy decoding, up to tiny floating-point effects.

Run the rule 200,000 times on the toy table and count what comes out. If the rule is right, the counts match the target column, not the draft column.

import numpy as np

rng = np.random.default_rng(7)
tokens = ["order", "refund", "parcel", "money"]
p = np.array([0.50, 0.25, 0.15, 0.10])  # large target model
q = np.array([0.30, 0.40, 0.20, 0.10])  # small draft model

residual = np.maximum(p - q, 0.0)
residual = residual / residual.sum()


def one_position():
    x = rng.choice(4, p=q)  # the draft proposes a token
    keep_prob = min(1.0, p[x] / q[x])
    if np.less(rng.random(), keep_prob):
        return x, True  # accepted
    return rng.choice(4, p=residual), False  # rejected: resample


n = 200_000
counts = np.zeros(4)
accepted = 0
for _ in range(n):
    x, ok = one_position()
    counts[x] += 1
    accepted += ok

for t, target, seen in zip(tokens, p, counts / n):
    print(f"{t:7s} target {target:.3f}  output {seen:.3f}")
print(f"acceptance rate: measured {accepted / n:.3f}, "
      f"sum of min(p, q) {np.minimum(p, q).sum():.3f}")

We ran this on Python 3.13 with NumPy 2.5.4. It takes about three seconds and prints the lines below. Your last digit may differ if you change the seed.

order   target 0.500  output 0.501
refund  target 0.250  output 0.249
parcel  target 0.150  output 0.151
money   target 0.100  output 0.100
acceptance rate: measured 0.800, sum of min(p, q) 0.800

What just happened? The output follows p, even though every guess came from q. The measured acceptance rate also equals the sum of min(p, q), here 0.30 plus 0.25 plus 0.15 plus 0.10.

That sum is the acceptance rate, alpha, for this spot. The first paper shows alpha is the sum of min(p, q), averaged over positions. In plain words, the more the two models agree, the more guesses survive.

We also ran the two wrong shortcuts on the same table. Always keeping the guess gave 0.300 for "order". Resampling from p after a rejection gave 0.402. Only the leftover rule gave 0.501, matching the target's 0.50.

On average, (1 minus alpha to the power k+1) divided by (1 minus alpha). This formula is from the first paper. It assumes each guess is kept independently with the same chance alpha.

With alpha at 0.8 and k at 4, that is 3.36 tokens per round. A round always gives at least 1 token and at most k+1. The script below checks the formula by playing whole rounds.

import numpy as np

rng = np.random.default_rng(11)
p = np.array([0.50, 0.25, 0.15, 0.10])  # large target model
q = np.array([0.30, 0.40, 0.20, 0.10])  # small draft model
alpha = np.minimum(p, q).sum()  # 0.80


def one_round(k):
    """Draft k tokens, verify them, return how many tokens this round adds."""
    for i in range(k):
        x = rng.choice(4, p=q)
        if not np.less(rng.random(), min(1.0, p[x] / q[x])):
            return i + 1  # i accepted drafts plus one resampled token
    return k + 1  # all k accepted plus one bonus token from the target


for k in (1, 2, 4, 8):
    rounds = 50_000
    measured = np.mean([one_round(k) for _ in range(rounds)])
    formula = (1 - alpha ** (k + 1)) / (1 - alpha)
    print(f"k={k}: measured {measured:.3f} tokens per round, formula {formula:.3f}")

k=1: measured 1.799 tokens per round, formula 1.800
k=2: measured 2.437 tokens per round, formula 2.440
k=4: measured 3.360 tokens per round, formula 3.362
k=8: measured 4.321 tokens per round, formula 4.329

What just happened? The measured averages match the formula to within about 0.01. Notice also the shrinking returns: doubling k from 4 to 8 adds only about one token per round, because long runs of kept guesses are rare.

It disappears when guesses are often wrong, when the draft is not cheap enough, when k is too long, or when big batches make the verify pass expensive. Each of these can push the gain below 1, which means slower than plain decoding.

The first paper gives the expected speed-up as tokens per round divided by (k times c plus 1). Here c is the time of one draft pass divided by the time of one target pass. The formula assumes the verify pass costs about one normal pass.

Low acceptance

When the draft often disagrees with the target, most rounds end after one or two tokens. At alpha 0.3, each round gives only 1.43 tokens, so the extra work barely pays. Gains also vary by task. The speculative sampling paper measured 2.46x on a code benchmark but about 1.9x to 2.0x on news summaries.

A draft that is not cheap enough

The draft runs k times per round, one pass after another. If one draft pass costs a fifth of a target pass, four guesses already cost 0.8 of a target pass. In the table, that turns 2.80x into 1.87x, and a weak draft into a slowdown.

Guessing too far ahead

A longer k means more draft passes, and the later guesses rarely survive. Our script found the best k for c equal to 0.05: 2 at alpha 0.3, 3 at alpha 0.5 and 8 at alpha 0.8. Past that, the gain falls again.

Large batches

A 2024 paper sums up the common view: speculative decoding helps mainly at small batch sizes. With many requests in one batch, each weight read already feeds lots of maths. The GPU moves towards being compute-bound, so the extra verify positions are no longer nearly free. Wasted guesses then cost real time.

Our script models this with a made-up verify cost. At alpha 0.8 and k 4, the speed-up drops from 2.80x to 1.53x if verifying costs two normal passes. It falls to 0.80x at four.

That same paper shows it is not a hard rule. It reports speed-ups at batch sizes of 32 to 256 for long contexts, using a draft with a trimmed KV cache. The KV cache is the stored attention data for each request.

At long contexts, reading that cache keeps decoding memory-bound even in large batches. vLLM's stable docs, as of 23 October 2026, describe the target case as medium-to-low traffic, memory-bound workloads.

Any cheap guesser works, because the accept rule only needs the guessed tokens and, ideally, their chances q. That is why several drafting styles exist. All of them keep the same verify step.

N-gram lookup shines when the answer repeats the prompt, such as editing code or quoting an order ID. In vLLM 0.32.0 it searches for the longest match within the prompt_lookup_min and prompt_lookup_max window, then proposes the tokens that followed.

Lookup has no real q, so vLLM 0.32.0's sampler treats the guess as certain. The rule then keeps it with chance p, and resamples from p with that token removed. We checked that this still gives exactly p.

Some methods guess a whole block of tokens in one draft pass instead of one at a time. That is called block or parallel drafting. SevyDevy has a separate guide on block drafting, linked at the end of this page.

Not every option keeps the exact guarantee. Some papers relax the accept rule to keep more guesses. In vLLM 0.32.0, the synthetic rejection mode accepts guesses at configured per-position rates and ignores p. It is meant for testing, so its output no longer follows the target.

You pass a speculative config when you start the server. As of 23 October 2026, the latest release is vLLM 0.32.0, published on PyPI on 18 October 2026. The keys below exist in its SpeculativeConfig source.

This config, copied from vLLM's docs example, turns on n-gram lookup. It guesses 4 tokens and matches the last 2 to 5 tokens. On the command line, put your model name right after vllm serve, then the flag with the JSON in single quotes. We could not run it here, because the sandbox has no GPU.

vllm serve your-model-name --speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_min": 2, "prompt_lookup_max": 5}'

For a separate draft model, set method to draft_model, set model to the draft model's name, and pick num_speculative_tokens. vLLM's docs example uses 5.

One default changes the maths. In vLLM 0.32.0, draft_sample_method is greedy, so model-based drafts propose their top token and q is treated as one-hot. The rule becomes keep with chance p, which is still exact.

Acceptance is then p of the guessed token, not the sum of min(p, q). On our toy table a greedy draft always guesses refund, so acceptance drops from 0.80 to 0.25. Setting draft_sample_method to probabilistic uses the full q rule, at extra GPU memory cost.

1. 1

   Measure the baseline first

   Run your real traffic without speculation. Record tokens per second per user and the time between tokens at your usual batch size.

2. 2

   Turn on one method

   Start with n-gram lookup if replies repeat the prompt. Otherwise try a draft model that shares the target's tokenizer.

3. 3

   Read the acceptance log

   vLLM logs a SpecDecoding metrics line by default. It shows the mean acceptance length, accepted and drafted counts, and per-position acceptance rates.

4. 4

   Compare at peak traffic too

   Repeat the measurement at your busiest batch size. A setting that wins at night can lose at noon.

5. 5

   Tune k, then decide

   Lower k if late positions are rarely accepted. Keep speculation only where the measured speed-up beats the baseline.

The metrics trap: two different acceptance rates

The "Avg Draft acceptance rate" in vLLM's log is accepted tokens divided by drafted tokens. It is not the paper's alpha. Guesses after the first rejection count as drafted but can never be accepted.

We simulated vLLM's counters on the toy table with k at 4 and full q sampling. The true alpha was 0.800, but accepted over drafted came out at 0.589. Plugging 0.589 into the formula would badly underestimate the gain.

Use the mean acceptance length instead. vLLM computes it as 1 plus accepted divided by drafts, and our simulation gave 3.354. That matches the formula's 3.362 tokens per round, so it is the number to compare.

Frame it as a latency tool for memory-bound decoding, then show you know its price. Interviewers want the mechanism, the guarantee and the limits, in that order.

1. 1

   Name the bottleneck

   Decoding makes one token per target pass. At small batch sizes each pass is limited by reading weights from memory, so the chip has spare maths.

2. 2

   Describe the loop

   A cheap draft guesses k tokens. The target scores all of them in one pass. Each round adds between 1 and k+1 tokens.

3. 3

   State the guarantee

   Keep a guess with chance min(1, p/q), else resample from max(0, p minus q). The output distribution equals the target's, up to floating-point effects.

4. 4

   Give the maths

   Expected tokens per round are (1 minus alpha to the k+1) over (1 minus alpha). Divide by (k times c plus 1), the round's cost in target passes, to get the speed-up.

5. 5

   Say when you would turn it off

   Low acceptance, a slow draft, an overlong k, or large batches that make decoding compute-bound. Tune k per batch size.

6. 6

   Say how you would measure it

   Track mean acceptance length and time between tokens, at both quiet and peak traffic, against a no-speculation baseline.