DSpark Speculative Decoding Explained: Block Drafting and Confidence Checks

Every new token needs one full pass through the large model, so replies crawl out word by word. Each pass is slowed mainly by reading the model's weights from memory, not by the maths.

DSpark speculative decoding attacks this waste. It lets a cheap model guess several tokens ahead, and the large model checks all the guesses in a single pass.

Picture a common setup. A travel-booking app runs an open-weight model, one whose weights it can download, on its own GPUs with vLLM, an open-source serving engine, to answer chat questions. The setup is made up, but the bottleneck is real.

A customer types a question about trains to New Delhi on Friday. The answer is 200 tokens long, so the model makes 200 passes, one after another. At 30 milliseconds a pass (made up), that is 6 seconds of typing dots.

The original speculative decoding paper names the cause. Decoding is usually limited by how fast data moves through memory, not by arithmetic, so spare compute sits idle while weights stream in.

Six words carry the whole guide. Each one is used the same way from here on.

• Token: a small piece of text, often a word or part of a word. Models read and write tokens, not letters.
• Forward pass: one run of the model over its input. Plain decoding needs one pass for each new token.
• Target model: the large model whose answers you want. It has the final say on every token.
• Draft model, or drafter: a much smaller, cheaper model that guesses the next few tokens.
• Block: the group of tokens the drafter guesses in one go, for example 5 tokens.
• Acceptance rate: the share of guessed tokens the target model keeps. A rate of 0.8 means 8 in 10 guesses survive.

DSpark speculative decoding is a way to guess a whole block of tokens at once, keep the guesses accurate, and check only the guesses likely to survive. It comes from a July 2026 paper and now runs in vLLM.

It stacks three ideas. Each one fixes a problem the one before it leaves behind.

• Draft then verify: a small model guesses, the large model checks every guess in one pass. This is classic speculative decoding.
• Block drafting with a small fix: the drafter guesses all positions in one pass, then a tiny step lets each guess see the token before it.
• Confidence-scheduled verification: a score predicts how likely each guess is to survive, and a busy server checks only the guesses worth the cost.

The large model checks each guess with a probability rule, and that rule keeps the output distribution exactly the same as plain decoding. The original speculative decoding paper, first posted in November 2022, proves this, and DSpark uses the same rule.

The naive version, and why it is not enough

The naive idea is to keep a guess only if it matches the large model's single top choice. That works for greedy decoding, where the model always picks its top token. It breaks sampling, where the model picks tokens at random by their probabilities.

The real rule compares two probabilities for each guessed token: p from the large model and q from the drafter. Keep the guess with probability p divided by q, capped at 1.

• If the drafter was less sure than the large model (q at or below p), the guess is always kept.
• If the drafter was over-sure (q above p), the guess is kept only sometimes, in proportion to p over q.
• At the first rejected guess, a replacement token is drawn from the leftover probability, where p is larger than q. Every guess after it is thrown away.
• If every guess survives, the same pass already gives the large model's next prediction, so you get one bonus token free.

Think of a junior writer and a senior editor. The junior drafts the next few words of the reply, and the senior reads the whole draft in one sitting.

The junior maps to the draft model, and the senior maps to the target model. Reading in one sitting is the single forward pass. The senior crosses out from the first wrong word, writes the right word there, and drops the rest.

Where the picture breaks: a real editor judges right or wrong. The target model accepts with a probability, so a plausible but not top word can survive. That is what keeps the output distribution exact.

A toy you can run on a laptop

The code below needs only Python's standard library, no GPU and no model download. It fakes both models with small probability tables about trips to New Delhi, Navi Mumbai and Pune.

import random

VOCAB = ["to", "New", "Navi", "Delhi", "Mumbai", "Pune", "on", "Friday", "."]
BIG = {  # big model: chance of each next token, given the previous token
    "to": {"New": 0.45, "Navi": 0.35, "Pune": 0.20},
    "New": {"Delhi": 0.95, "Mumbai": 0.05}, "Navi": {"Mumbai": 0.95, "Delhi": 0.05},
    "Delhi": {"on": 0.7, ".": 0.3}, "Mumbai": {"on": 0.7, ".": 0.3},
    "Pune": {"on": 0.7, ".": 0.3}, "on": {"Friday": 1.0}, "Friday": {".": 1.0}, ".": {"to": 1.0},
}

def big(prev):
    return [BIG[prev].get(t, 0.0) for t in VOCAB]

def small(prev):  # draft model: mostly right, 20 percent spread over every token
    return [0.8 * p + 0.2 / len(VOCAB) for p in big(prev)]

def pick(probs, rng):
    return rng.choices(VOCAB, weights=probs)[0]

def step(prev, k, rng):
    drafts, ctx = [], prev
    for _ in range(k):  # 1. the small model guesses k tokens
        q = small(ctx)
        tok = pick(q, rng)
        drafts.append((tok, q))
        ctx = tok
    out, ctx = [], prev
    for tok, q in drafts:  # 2. the big model checks every guess
        p, i = big(ctx), VOCAB.index(tok)
        if rng.random() < min(1.0, p[i] / q[i]):
            out.append(tok)  # keep the guess
            ctx = tok
            continue
        leftover = [max(0.0, a - b) for a, b in zip(p, q)]
        out.append(pick(leftover, rng))  # 3. fix the first wrong guess, stop
        return out
    out.append(pick(big(ctx), rng))  # all kept: one bonus token for free
    return out

rng, prev, total = random.Random(42), "to", 0
for n in range(1, 6):
    out = step(prev, 4, rng)
    total += len(out)
    print(f"pass {n}, {len(out)} token(s): {' '.join(out)}")
    prev = out[-1]
print(f"5 big-model passes, {total} tokens")

Run it with python3 on Python 3.13; we tested 3.13.16. You should see exactly this output, because the random seed is fixed:

• pass 1, 2 token(s): Navi Mumbai
• pass 2, 1 token(s): on
• pass 3, 5 token(s): Friday . to New Delhi
• pass 4, 2 token(s): on Friday
• pass 5, 3 token(s): . to Navi
• 5 big-model passes, 13 tokens

We also ran 20,000 steps of a fuller version of this simulation to check the lossless claim. After the word to, the large model's table says New 0.45, Navi 0.35 and Pune 0.20. The speculative output gave 0.443, 0.353 and 0.204.

Each pass of the large model gives on average between 1 token and the block size plus one, depending on the acceptance rate. The original paper gives a formula for this when every guess has the same chance of surviving.

The formula is (1 - a^(g+1)) / (1 - a). Here a is the acceptance rate and g is the number of guessed tokens.

Read the first row slowly. At 0.5 acceptance, guessing 8 tokens instead of 4 moves you from 1.94 to 2.00 tokens, because the chance of reaching the eighth guess is tiny. Long blocks only pay when guesses are good.

This is also why tokens per pass is not the same as speed-up. Drafting costs time, and checking 8 guesses costs more than checking 2 when the GPU is busy.

Watch the stats line vLLM logs for speculative decoding, or the matching Prometheus counters. In v0.32.0 the line starts with SpecDecoding metrics and appears every 10 seconds unless you pass --disable-log-stats.

• Mean acceptance length: average tokens gained per step, including the bonus token. vLLM computes it as 1 plus accepted tokens divided by drafts.
• Per-position acceptance rate: one number per block position. Expect it to fall from the first position to the last.
• Avg Draft acceptance rate: accepted draft tokens as a percentage of drafted tokens.
• Prometheus counters: vllm:spec_decode_num_drafts, vllm:spec_decode_num_draft_tokens and vllm:spec_decode_num_accepted_tokens. Each is exported with a _total suffix.

For the travel chatbot, a mean acceptance length of, say, 3.4 means each pass of the large model yields 3.4 tokens on average. Compare tokens per second per user with speculative decoding on and off, on the same traffic.

Also check quality did not move. When vLLM added the online estimator, its pull request reported GSM8K maths accuracy, pooled across its test models, of 95.51 percent with adaptive verification off and 95.43 percent on. The authors called that no significant change.

A block drafter guesses every position in one pass, so position 2 cannot see what position 1 chose. When the right second word depends on the first, the guesses stop fitting together.

Classic drafters are autoregressive: they guess one token, feed it back, and guess the next. That keeps guesses coherent but costs one drafter pass per token. Block drafters, which the DSpark paper calls parallel drafters, pay for one pass but lose that link.

Our simulation answers it. Position 2 cannot see position 1, so its best guess is the average: Delhi 0.50, Mumbai 0.50, whatever position 1 said. Out of 10,000 drafts, 50.6 percent paired New with Mumbai or Navi with Delhi.

The chance that the second guess survives the check was 0.55. That chance is the overlap of the two distributions: add up, over every token, the smaller of the drafter's and the large model's probability. The DSpark paper trains its confidence head to predict this same overlap.

The DSpark fix: a tiny step that looks back one token

DSpark keeps the parallel drafter as its backbone and adds a lightweight sequential head on top. The default version is a Markov head, Markov meaning it looks back only one step. It adjusts each position's scores using only the token guessed just before it.

The head has no attention layers: it looks up the previous token and applies one small matrix multiply. The paper measured it adding 0.2 to 1.3 percent to round latency over a plain parallel drafter, for blocks of 4 to 16 tokens.

In our toy, letting position 2 see position 1, with a slightly-off 0.90 and 0.10 guess, raised its survival chance from 0.55 to 0.95. The paper's own maths, code and chat benchmarks used a block of 7 tokens. Average accepted length rose 16.3 to 18.4 percent over a parallel drafter, and 26.7 to 30.9 percent over an autoregressive drafter.

In the authors' live serving system, per-user generation was 60 to 85 percent faster at matched throughput than their previous production drafter. These are the authors' numbers on their own system; we have not reproduced them.

It scores every guess by its chance of surviving, then checks the highest-scoring guesses across all requests, stopping at the amount that gives the most tokens per second. Weak guesses are dropped before the large model pays for them.

Checking is nearly free when the GPU has spare room. Under heavy load, every rejected guess takes batch space a real token could have used. The DSpark paper names this waste as the second problem it solves.

Survival probability

A confidence head predicts, for each position, the chance the guess survives if every earlier guess survived. Multiply those chances along the block to get each position's survival probability.

If position 1 has 0.9 and position 2 has 0.8, position 2 survives with 0.9 times 0.8, which is 0.72. Survival can only fall along a block, so trimming always cuts from the end.

Picking the budget

vLLM's v0.32.0 source sorts every request-and-position slot by survival. It then picks the number of slots that maximises expected accepted tokens divided by the step's estimated cost in milliseconds.

Expected accepted tokens are one guaranteed token for each generating request plus the sum of the chosen survival scores. The cost comes from tables vLLM measures at startup, using a synthetic context of 8,192 tokens by default.

We copied that objective into our toy with a made-up cost of 25 milliseconds plus 0.1 per token checked. With 16 requests, it kept 62 of 80 guesses. With 128 requests, it kept 324 of 640 and raised toy throughput from 4.12 to 4.96 tokens per millisecond.

Confident requests kept 4 guesses each at 128 requests, and the least confident kept 1. The scheduler shifts checking towards requests where it pays.

Two details the docs do not spell out

First, the scores are slightly old. The paper says production ranks using confidences from two steps earlier, because the batch size must be fixed before the current draft finishes. vLLM's code also reads a stored earlier step's confidences when sizing the budget.

Second, the confidence does not have to come from a trained head. Since v0.30.0, vLLM fits a logistic model, a simple formula that turns a score into a probability, from real accept and reject results. Its input is how sure the drafter was of its top token, with one shared slope and one offset per position.

On one large model with a DSpark drafter and about 4.5 million graded guesses, the vLLM blog reports AUC 0.8336 for the estimator and 0.8191 for the trained head. The head was better at position 1. AUC measures how well a score separates guesses that survive from guesses that do not.

Pass a speculative config to vllm serve with method set to dspark and enable_adaptive_verification set to true. You also need a DSpark drafter checkpoint trained for your target model.

Here is the JSON you pass with the --speculative-config flag. We read every key in the vLLM v0.32.0 source; we could not run it, because it needs a GPU and a real drafter.

{
  "method": "dspark",
  "model": "PATH_TO_DSPARK_DRAFTER",
  "num_speculative_tokens": 5,
  "draft_sample_method": "probabilistic",
  "enable_adaptive_verification": true
}

What blocks it from starting

• Full CUDA graphs are required, so the --enforce-eager flag is rejected at startup.
• LoRA adapters are not supported with adaptive verification.
• Pipeline parallelism is not supported with adaptive verification.
• The attention backend must handle query lengths decided on the GPU. Backends that cannot are rejected with an error naming them.
• For drafters other than DSpark, which rely on the estimator, the use_local_argmax_reduction option is rejected. The estimator needs the drafter's full scores.

If your prompts are much longer than 8,192 tokens, raise the profiling context with the environment variable VLLM_ADAPTIVE_VERIFICATION_PROFILE_CONTEXT_LEN. The vLLM docs give 131072 as an example.

It stops helping when guesses are mostly wrong or the GPU is already full of real work. Both cases spend compute on tokens that get thrown away.

• Low acceptance: free-form chat accepts fewer guesses than maths or code in the DSpark paper's own benchmarks. The drafter must suit your traffic.
• Heavy load without trimming: checking a full block for every request crowds out real tokens. Adaptive verification exists for this case.
• Autoregressive drafters: the vLLM blog says EAGLE-style and MTP drafters gain less from trimming, because they already paid to draft the guesses that get dropped.
• Tuned fixed lengths: the same blog notes a smaller fixed block sometimes beat adaptive verification at certain concurrencies. The docs pitch it as one setting for every load, so measure both.

Frame it as a latency fix for decoding that keeps outputs identical in distribution, then show you know its cost under load. Use the travel chatbot as your example.

1. 1

   State the bottleneck

   Decoding makes one pass of the large model per token and is limited by memory bandwidth. Spare compute sits idle at low batch sizes.

2. 2

   Explain draft and verify

   A small drafter guesses a block of tokens. The large model checks them all in one pass, keeping guesses up to the first rejection.

3. 3

   Show the output does not change

   Name the accept rule: keep a guess with probability p over q, capped at 1, and resample from the leftover on rejection.

4. 4

   Size the gain

   Use acceptance rate and block length. At 0.8 acceptance and 4 guesses, expect about 3.4 tokens per pass.

5. 5

   Add block drafting

   A parallel drafter guesses all positions in one pass but loses coherence. DSpark adds a previous-token head to restore it cheaply.

6. 6

   Handle heavy load

   Score each guess's survival, rank across requests, and verify only the slots that raise tokens per millisecond on a measured cost table.

7. 7

   Name the checks

   Monitor mean acceptance length, per-position acceptance and per-user tokens per second, and run an accuracy check with the feature on and off.