SGLang's October 2026 KV Cache Exposure Explained: Stale Memory Between Users

It hands one user's private text to another user, and nothing looks broken. The reply arrives with a normal success code, reads fluently and contains someone else's data.

That is what the SGLang KV cache exposure did. SGLang is an open-source engine that runs large language models on GPUs. Under one specific set of conditions, a request could read memory still holding earlier requests' data.

Picture a common setup. A team self-hosts a large open-weight model with a very long context window, and splits the work across two groups of GPU servers to serve more users. The setup is made up, but the failure is the one Cloudflare reported.

One day a user pastes a huge document, longer than the memory the first group of servers has room for. Instead of an error, the user gets a confident answer that quotes a stranger's conversation.

How did a stranger's conversation end up in that answer? Eight terms explain it, and each one comes back in the next three sections.

• Token: a small piece of text, about a word or part of a word. Models read and write tokens, and limits are counted in tokens.
• Context window: the longest input, in tokens, a model is configured to accept in one request.
• KV cache: the model's working memory for a request. KV stands for key and value, two lists of numbers the model stores for every token so it does not recompute them.
• Page: a fixed-size block of KV cache memory. The server hands pages to requests and takes them back when requests end.
• KV pool: all the pages one server has. Its size in tokens is the most context it can actually hold.
• Prefill: the first phase of a request, where the model reads the whole prompt and builds its KV cache.
• Decode: the second phase, where the model writes the answer one token at a time while reading that KV cache.
• Disaggregated serving: running prefill and decode on separate servers and copying the KV cache between them over a fast network link. SGLang calls this PD disaggregation.

The SGLang KV cache exposure was a cross-request data leak in SGLang's disaggregated prefill and decode mode. It hit one model Cloudflare served on its hosted inference product, and an outside researcher reported it.

Cloudflare's post-mortem says 155 responses were affected, all to prompts longer than 681,280 tokens. The exposure ran from the model's launch on 27 August 2026 until 9 September 2026.

Cloudflare says it found no evidence of deliberate exploitation. It also says no other model it hosts was affected. Both statements are Cloudflare's own findings about its own service.

The report took seven weeks to become public because it was a coordinated disclosure. Cloudflare and the SGLang maintainers fixed the engine first, then explained it.

Two problems met. A configuration gap let a prompt be too big for the prefill servers, and an engine bug sent that oversized request onward instead of refusing it.

Here is an everyday picture before the details. Freed KV pages are like hotel rooms re-let without cleaning. The front desk turned a guest away but still handed over the keys. Its check asked only whether the stay had ended, not whether it had been refused.

That turned-away guest walks into rooms full of the last guest's papers and reads them out loud. Where the picture breaks: a hotel guest would notice the mess, while a model simply reads whatever memory it is given and never knows it is not its own.

The configuration gap

The model accepted prompts up to 1,048,576 tokens. The prefill servers' KV pool held only 681,280 tokens. Any prompt between those two numbers was allowed in but could not fit.

Cloudflare's fix for this part was simple. It matched the settings of an earlier model, so each prefill pool now holds about 2.04 million tokens.

The swapped prompt

When SGLang refuses a request, it does not stop it at once. It swaps the long prompt for a single token, called a stub, so the refused request stays cheap. In the 0.5.22 source this happens in a function named set_finish_with_abort, so the code calls a refusal an abort.

The refusal is written down as a note to act on later, in a field named to_finish. The request is not marked finished yet.

The check that asked the wrong question

Every queue that admits requests asked one question: is this request finished? That answer comes from a different field, finished_reason, which was still empty. So the refused request looked alive and passed every gate.

Cloudflare puts it plainly: the queues checked whether a request was finished, not whether it was rejected. The 0.5.22 source carries a comment saying the same thing about to_finish.

Decode reads pages nobody wrote

Prefill built KV cache for the one-token stub and sent it on. The decode server had reserved pages for the full prompt length, but only the stub's data arrived.

That copy uses RDMA, which means remote direct memory access. One machine writes straight into another machine's GPU memory without asking its CPU. Cloudflare used a transfer library called Mooncake for this.

The other reserved pages were not empty. They still held KV cache from earlier requests, because freed pages are reused without being wiped. Decode read them as if they were the prompt and wrote an answer from them.

Because every signal a team usually watches said all was well. Cloudflare's report lists three of them.

• The status code was HTTP 200, which means success, instead of HTTP 400, which means a bad request.
• The prompt_tokens count in the response matched the long prompt, so usage numbers looked right.
• The reply was coherent. Language models write fluent text from whatever memory they read.

Cloudflare says it knew before launch that the pool was smaller than the context window. It expected the engine to refuse prompts it could not hold. Its long-context tests only checked HTTP codes and engine speed, so they passed.

The bug was not new, either. Cloudflare notes that another user had reported it to SGLang as issue 30233 on 6 July 2026. That report described decode reserving space for a full prompt but receiving one token of KV data.

One clue did exist. Cloudflare notes the leaked replies arrived faster than reading such a long prompt should take. A system that tracks time per prompt token could have flagged that.

You can run a small toy in plain Python. It is not SGLang code. It copies the shape of the bug: a shared page pool, pages reused without wiping, and a refusal that does not stop the request.

PAGE = 4  # tokens per page (toy size)
pool = [None] * 8  # shared KV pages on the decode server
free = list(range(8))

def alloc(n):
    return [free.pop(0) for _ in range(n)]

def release(pages):
    free[:0] = pages  # freed pages are reused next; old data is NOT wiped

# Request A from customer 1 runs normally and finishes.
a = alloc(2)
pool[a[0]], pool[a[1]] = "A: card ends 4417", "A: ship to Pune"
release(a)

def serve(prompt_tokens, prefill_capacity, guard):
    if prompt_tokens > prefill_capacity:
        if guard:
            return "HTTP 400: prompt too long"
        sent = ["placeholder"]  # prompt swapped for a one-token stub
    else:
        sent = [f"B page {i}" for i in range(-(-prompt_tokens // PAGE))]
    pages = alloc(-(-prompt_tokens // PAGE))  # decode reserves the full length
    for slot, data in zip(pages, sent):
        pool[slot] = data  # only len(sent) pages are actually written
    seen = [pool[p] for p in pages]
    release(pages)
    return f"HTTP 200, decode reads {seen}"

print("old path:", serve(prompt_tokens=8, prefill_capacity=6, guard=False))
print("fixed path:", serve(prompt_tokens=8, prefill_capacity=6, guard=True))
print("normal request:", serve(prompt_tokens=8, prefill_capacity=16, guard=True))

Save it as toy_kv_pool.py and run python3 toy_kv_pool.py with Python 3.8 or later. You should see exactly these three lines.

old path: HTTP 200, decode reads ['placeholder', 'A: ship to Pune']
fixed path: HTTP 400: prompt too long
normal request: HTTP 200, decode reads ['B page 0', 'B page 1']

The first line is the leak. Request B was too long, so only its stub was written, and the second reserved page still held request A's shipping city. The status still says 200.

The second line refuses the request up front, which is what 0.5.21 added. The change that closed the real leak, in 0.5.20, works one step later: prefill retires the refused request instead of handing it to decode. The third line shows a normal request writing every page it reads.

Run SGLang 0.5.22 or later, as Cloudflare recommends. The leak path itself closed in 0.5.20, 0.5.21 turned on safer defaults, and 0.5.22 fixed a failed-send case that 0.5.21 still had. We checked each released wheel's source on 27 October 2026.

The deferred-release column is the fix for a second, related race, and the checksum is an extra guard; both come next. Mooncake, NIXL and MORI are the transfer libraries SGLang supports for copying KV cache between servers.

The change that closed the leak is pull request 36988. In 0.5.20, when prefill finishes a refused request, it retires the request instead of sending its last KV chunk to decode. In 0.5.19 the same code path still sends it.

Pull request 38935, first in 0.5.21, refuses such a request even earlier, before any transfer is set up. Its own description calls it a waste-and-latency fix, not a correctness fix. The 0.5.22 source calls a helper named is_unadmitted_reject for this.

Cloudflare's post says the patch shipped in 0.5.20 with follow-ups in 0.5.21 and 0.5.22, which matches our source check. Its timeline gives 2 October for 0.5.21 and 19 October for 0.5.22, one to three days after the wheels reached PyPI.

If you upgrade to 0.5.22, check your Python first. PyPI lists Python 3.11 or later for 0.5.22, while 0.5.21 accepted 3.10.

It is a timing bug where a copy finishes after the receiver has already given the memory away. It is related to the incident but separate from the over-long prompt path.

Here an abort means a request cancelled in the middle of a KV transfer, for example when the client disconnects. It is not the over-long refusal from earlier. RDMA writes for that request may still be on their way. The SGLang team writes that if decode releases and reassigns those slots before the writes complete, the late writes can overwrite another request's KV cache.

Deferred release fixes this by waiting. In the 0.5.22 source, an aborted request's pages are held until prefill confirms the transfer drained, or until a timeout, which defaults to 30 seconds.

A write that arrives after the timeout could still land in reused pages, so the timeout is a margin, not a guarantee. The SGLang post suggests 60 to 120 seconds through SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT for a wider margin.

The SGLang post scopes this race to the cross-engine transfer path. It says a single-instance deployment without that path is not affected by this specific race. Default PD configurations were affected from the 0.4 line until 0.5.21 (0.5.22 for MORI).

It adds that deployments using the MORI transfer backend need a build that includes pull request 37077, which is in 0.5.22. The post also says failed Mooncake sends could still be confirmed as drained until pull request 43719, which is in 0.5.22. It lists the same fix for NIXL and MORI, pull request 45973, for 0.5.23, which was not on PyPI on 27 October 2026.

If you are stuck on 0.5.19 or 0.5.20 for now, the SGLang post says to set SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1 on both servers. Both versions have the setting, but it is off by default. The post says it covers only aborts started by decode, the most common kind, so treat it as a stopgap until you upgrade.

Check the version, compare pool size with context length on every server, and test answers for correctness at your longest allowed prompt.

1. 1

   Check the installed version

   Run pip show sglang on every prefill and decode server. Move to 0.5.22 or later, which includes the failed-send fix for Mooncake. NIXL and MORI users should also watch for 0.5.23.

2. 2

   Read the pool size from the startup log

   SGLang logs a line with max_total_num_tokens and context_len at startup. Find it in each server's log and write both numbers down.

3. 3

   Compare the two numbers

   If max_total_num_tokens is smaller than context_len, a prompt can be accepted that cannot fit. Lower the context length or give the pool more memory.

4. 4

   Watch the same number as a metric

   Start servers with --enable-metrics and read the gauge sglang:max_total_num_tokens. Re-check it after any change to weights, quantisation, parallelism or memory settings.

5. 5

   Turn on the KV checksum where it fits

   Pass --disaggregation-enable-kv-checksum to both prefill and decode servers, from 0.5.21. It compares an Adler-32 checksum of each request's KV pages, and a mismatch aborts the request.

6. 6

   Test output, not just status

   Send a prompt near your maximum length with a unique marker sentence, and check the answer refers to it. A 200 code with an unrelated answer is a failure.

For the log check, a quick search is enough. You expect one line per server process with both numbers in it.

pip show sglang | grep Version
grep -o 'max_total_num_tokens=[0-9]*.*context_len=[0-9]*' server.log

A healthy result has the first number larger than the second, for example max_total_num_tokens=2040000 next to context_len=1048576. These values only echo the sizes in Cloudflare's report; read your own numbers from your log.

The checksum has limits. The SGLang post says both engines need the same KV layout and that it costs an extra GPU read and a sync point per transfer. Cloudflare says such checks do not yet cover every model architecture.

Cloudflare recommends turning on corruption detection where your model supports it, naming this checksum and a second check, --kv-canary. In the 0.5.22 source that flag takes none, log or raise, and defaults to none; its help text calls log production-safe. Treat both as extra guards on top of the upgrade.

Treat memory shared between users as a security boundary, and test it like one. The lessons apply to any engine that pools KV cache across requests.

• Refuse early and loudly. A request that cannot be served should return an error before it holds any resource, not carry a stored refusal through the system.
• Keep one source of truth for request state. Two fields that both mean done, read by different code, is how this bug slipped through.
• Do not trust success codes in tests. Check that answers match prompts, especially at the longest lengths you allow.
• Never assume an engine fails safely at a limit. Cloudflare knew the pool was smaller than the context window and expected a clean refusal; test that refusal directly.
• Add layers that make leaks impossible, not just unlikely. Cloudflare prototyped ownership tags on pages at under 1% throughput cost, disabled them over reliability problems and plans a production version.

Cloudflare also plans to model its KV transfer protocol formally, which means proving properties with a tool rather than testing examples. That work is described as future, not done.

Use it as a story about isolation in shared memory. Interviewers want to hear that you see the KV cache as user data, not just a speed trick.

1. 1

   Name the shared resource

   In LLM serving, KV cache pages are pooled across users for speed. Freed pages are reused without wiping, so stale data is usually present.

2. 2

   State the invariant

   A request may only read pages written for that request. Every path, including errors and aborts, must keep this true.

3. 3

   Show how it broke

   A rejected over-long request was not stopped, so decode read reserved pages that held other users' data.

4. 4

   Give layered fixes

   Never hand a refused request onward, reject it at intake, hold pages until transfers drain, verify transfers with checksums, and size pools to the context window.

5. 5

   Add detection

   Test outputs for correctness at maximum length, and alert on replies that are impossibly fast for their prompt size.