vLLM Preload Explained: Restart LLM Servers in Seconds with a Weight Cache

A restarted LLM server answers nothing until it has rebuilt what it held in GPU memory. The vLLM docs put that at minutes, for reading weights from disk and processing them for quantization. vLLM preload keeps the costliest part, the model weights, alive across restarts.

Picture a common setup. A support team runs a voice bot that answers customer calls for online shops in Telugu, Hindi and English. It serves one model on two vLLM servers, each with 4 GPUs. The setup is made up, but the problem is real.

Each server rents for Rs 1,400 an hour, and every restart of the serving process takes about 7 minutes (made up). Setting changes and crashes cause about 60 restarts a month. That is 7 hours, or Rs 9,800, of GPU time that serves no calls.

Worse, while one server restarts, the other carries every call. The team wants restarts that cost seconds, not minutes.

You need seven terms. Each has a plain meaning.

• Weights: the learned numbers that make up a model, stored in files on disk, usually in the safetensors format.
• Tensor: one named block of weights, such as one grid of numbers inside a layer.
• Engine: the vllm serve process that loads the model and answers requests. This is the part you restart.
• Quantization: storing weights in fewer bits to save memory. vLLM runs a post-processing pass on such weights after loading them.
• Tensor parallelism (TP): splitting each layer across GPUs, so each GPU, called a rank, holds one shard (slice) of the weights.
• Daemon and Unix socket: a background program, and a private channel to it that works only inside one machine, like an office phone extension.
• CUDA IPC handle: a token that lets one process use another process's GPU memory without copying it. CUDA and ROCm are the GPU platforms of NVIDIA and AMD.

vLLM preload keeps model weights resident in GPU memory across engine restarts. A long-lived daemon owns them, and each new engine borrows them.

Think of a restaurant kitchen that keeps its stock simmering overnight. The stock is the weights, the night cook who tends the pot is the daemon, and the morning cook is the restarted engine. Boiling bones for hours is the slow disk load and quantization pass.

Where the picture breaks: in the default mode the morning cook does not take a portion. They use the night cook's pot directly, so the night cook must stay on shift for as long as the morning cook works.

The daemon loads once and waits

The vllm preload command starts one daemon per GPU. Each loads its shard from disk, runs quantization post-processing and exports every weight as a CUDA IPC handle. It opens its Unix socket only once the shard is cached.

The engine maps instead of loading

An engine started with --load-format ipc_cache builds an empty model shell with no memory behind it. It then asks its GPU's daemon for the weights. In the default zero_copy mode, it maps the daemon's memory directly. The docs say this cuts restarts from minutes to seconds.

Preload removes the two slow jobs the docs name: reading weights from disk and quantization post-processing. The docs say preload covers only model weights today, so the rest of startup still runs. Seconds does not mean zero.

The team's test restart now takes about 40 seconds instead of 7 minutes (made up). The rest is startup work that preload does not cover. Beginners can stop here: you know what preload keeps, what it skips and why restarts get shorter.

Before serving, the engine compares its fingerprint with the daemon's. A fingerprint is a short summary of everything that shapes the weights. By default, any difference sends the engine to disk: slow but correct.

In v0.31.0 the fingerprint holds a hash of the checkpoint files, the model architecture and revision, and the TP and data-parallel sizes and ranks. It also holds dtype (the number format), the quantization method and settings, and the vLLM version. A separate GPU ID check stops a stale socket serving the wrong GPU.

What the checkpoint hash does not see

Pause and predict: the team retrains the model on new call transcripts, keeping every layer the same size. Does the fingerprint change?

No. The docs say the checkpoint is hashed from safetensors metadata, so identical weights in two folders still match. That metadata holds tensor names, number formats, shapes and byte offsets, never the numbers themselves.

We ran vLLM's own hash function on v0.31.0 and v0.32.0. Two checkpoints with the same shapes and different values gave the same hash, even in different folders. A different shape gave a different hash.

The revision field does not save you either: it is a model hub's version label, and for a local folder it stays empty unless you set --revision. So a daemon started before the retrain hands the old weights to the new engine, even from a new folder, with no error. The one exception we found: if the new files carry different optional metadata, the hash changes.

The toy code below copies that logic, including the blind spot. It runs on plain Python 3.

# Toy model of the weight-cache fingerprint check (not vLLM's real code)
import hashlib, json

def checkpoint_hash(tensors):
    # Like vLLM: hash tensor names, dtypes and shapes, never the numbers
    header = {name: [t["dtype"], t["shape"]] for name, t in tensors.items()}
    return hashlib.sha256(json.dumps(header, sort_keys=True).encode()).hexdigest()[:8]

def fingerprint(tensors, **settings):
    return {"checkpoint": checkpoint_hash(tensors), **settings}

def pick_loader(daemon_fp, engine_fp, fallback=True):
    if daemon_fp is None:                      # socket not open yet
        changed = ["daemon not ready"]
    else:
        changed = [k for k in engine_fp if daemon_fp[k] != engine_fp[k]]
    if not changed:
        return "ipc_cache: map the daemon's weights"
    if not fallback:
        raise RuntimeError(f"weight cache refused: {changed}")
    return f"disk: slow load, because of {changed}"

settings = {"architecture": "MyModelForCausalLM", "tp_size": 4, "tp_rank": 0,
            "dtype": "bfloat16", "quantization": "fp8", "vllm_version": "0.31.0"}
old = {"layer0.weight": {"dtype": "BF16", "shape": [4096, 4096], "values": "v1"}}
retrained = {"layer0.weight": {"dtype": "BF16", "shape": [4096, 4096], "values": "v2"}}

daemon = fingerprint(old, **settings)
print(pick_loader(daemon, fingerprint(old, **settings)))         # crash restart
print(pick_loader(daemon, fingerprint(old, **{**settings, "quantization": "fp4"})))
print(pick_loader(daemon, fingerprint(retrained, **settings)))   # retrained weights

What just happened? The crash restart maps the cached weights. The 4-bit build takes the disk path because the quantization changed. The retrained weights also map, because only the values changed, which is exactly the trap above.

What fallback false really does

In v0.31.0 source, fallback false does two different things. If the socket is not open yet, the engine waits up to 300 seconds for the daemon, retrying every half second, then fails. If the fingerprint differs, it fails at once.

The team keeps fallback on, so a missing daemon never blocks a restart. It checks the engine log instead, and restarts the daemons itself when it ships new weights.

Use the default zero_copy mode unless the engine must not depend on the daemon. Set the mode in --model-loader-extra-config.

Copy mode has a catch the table hides. Once the daemon releases its cache, it answers later requests with an error, so the next restart loads from disk. Copy mode suits a one-time handoff, not repeated fast restarts.

Start the daemons with the same model and parallel settings as the engine, then start the engine with the ipc_cache load format. The daemons must load from disk, so passing ipc_cache to vllm preload is an error.

Start the daemon, check it, then start the engine

# 1. Start the weight-cache daemons once; they load from disk.
#    vllm preload stays in the foreground until stopped, so give it its own
#    terminal or run it as a service (for example under systemd).
vllm preload --model /models/support-bot --tensor-parallel-size 4 \
  --weight-cache-health-port 8099 --weight-cache-health-host 127.0.0.1

# 2. In a second terminal, wait for the daemons: /health answers 503 until
#    every GPU's shard is cached, then 200 (curl -f treats 503 as a failure)
until curl -sf -o /dev/null http://127.0.0.1:8099/health; do sleep 5; done

# 3. Start (or restart) the engine; it maps weights from the daemons.
#    It also stays in the foreground, so run step 4 from another terminal.
vllm serve /models/support-bot --tensor-parallel-size 4 \
  --load-format ipc_cache 2>&1 | tee engine.log

# 4. Check the engine did not quietly load from disk (no output is good)
grep "falling back to disk loading" engine.log

# Optional engine flag: copy mode, and no quiet fallback to disk
#   --model-loader-extra-config '{"mode": "copy", "fallback": false}'

The /health port is off by default and binds to 0.0.0.0 when enabled, so the team binds it to 127.0.0.1. It returns HTTP 200 once every daemon is serving and 503 before that, or after any daemon dies.

The grep checks for vLLM's warning when the cache is unusable. An engine that connects before a shard is cached loads from disk, so step 2 waits for 200 and the daemons start at boot. We had no GPU to run these commands end to end. We checked the preload flags against the v0.31.0 and v0.32.0 parsers, and the wait loop against a stand-in server.

If you need to turn off the FlashInfer autotune pass, check the flag name. As of 19 October 2026, the v0.32.0 release notes mention --preload-autotune, and the developer docs mention --disable-flashinfer-autotune.

We ran the v0.31.0 and v0.32.0 argument parsers: both reject those two flags. The switch that works is --no-enable-flashinfer-autotune, and the docs shipped inside the v0.32.0 package name it.

Restart during a rolling deploy or after a crash

A rolling deploy restarts servers one at a time while the others keep serving. Preload shortens each gap.

1. 1

   Drain one server

   Stop new calls to it at the load balancer, the router that spreads calls across servers, and let open calls finish.

2. 2

   Restart only the engine

   Restart vllm serve with the new settings. Leave vllm preload running, since stopping it frees the cached weights.

3. 3

   Check it used the cache

   Run the grep from the setup. A fallback warning, or a start as slow as a cold one, means the engine loaded from disk.

4. 4

   Return it to the pool

   Add it back once health checks pass, then repeat on the next server.

5. 5

   Treat crashes the same way

   A supervisor such as systemd restarts vllm serve. The daemons are separate processes, so they still hold the weights.

The team's 60 monthly restarts now cost about 40 minutes of GPU time, or Rs 930, instead of Rs 9,800 (made up). The other server is overloaded for 40 seconds, not 7 minutes.

Preload needs the same user, software and model setup on each machine. These limits come from the docs and the source, as of 19 October 2026.

• CUDA and ROCm only. On any other platform the engine raises an error, even with fallback on. Our CPU-only test machine raised it on v0.31.0.
• Every quantization method in the model must support pre-processed weights. If one does not, vllm preload fails while loading the model. An engine with fallback on then finds no daemon and loads from disk.
• The daemon and engine must share a machine and a Linux user. CUDA IPC handles are node-local, so each machine runs daemons for its own GPUs.
• Both need the same vLLM version, model, dtype, quantization and parallel layout, or the fingerprint check sends the engine to disk.
• Tensor, expert and data parallelism are supported, including across machines, with one vllm preload launcher per machine. Expert parallelism spreads a mixture-of-experts model's experts over GPUs; data parallelism runs full copies side by side.
• Pipeline parallelism, which splits layers into stages on different GPUs, is rejected by v0.31.0. The v0.32.0 code accepts it, but the v0.32.0 docs still say rejected in two places.
• Speculative decoding uses a small draft model to guess tokens that the main model then checks. Preload also caches the drafts for MTP, where the main model can itself predict several tokens at once, and for EAGLE and EAGLE3 draft models. Other draft types load from disk.

The team fits: CUDA GPUs, one machine per model copy and tensor parallelism only.

Only among trusted processes of the same user on one machine. The docs say its protocol uses pickle and is meant only for those processes.

Pickle is Python's built-in way to turn objects into bytes, and loading pickle from an attacker can run their code. vLLM limits access with an owner-only folder (mode 0700) and owner-only socket files (0600).

Both sides refuse symlinked or foreign-owned sockets. On Linux, the daemon also checks each caller's user ID.

Preload helps only when an engine restarts where a daemon already holds matching weights. Every first load still comes from disk. The table applies that rule; it is our reasoning, not the docs'.

So preload does not fix slow autoscaling. A third server added for a festive sale still needs 7 minutes (made up), so the team starts it early.

Not yet: they are an experimental feature added in v0.31.0. CRIU, a Linux tool that freezes and restores processes, saves the whole initialized engine to disk. Restore reloads the model files and the KV cache before opening its HTTP port. The KV cache is GPU memory where the engine keeps working data for each request's past tokens.

Restore checks a one-token test answer and never silently falls back to a normal start. It needs Linux on x86-64, the common PC and server processor type, one NVIDIA GPU, tensor parallelism of 1, and root or passwordless sudo.

It also needs a remote model ID pinned to a fixed revision; local model folders are not supported. Restore needs the same host, GPU, driver, vLLM version, model revision and settings. Snapshots can approach the size of the captured memory.

The pull request's author reported one test on an earlier source commit: a 32-billion-parameter BF16 (16-bit) model on one 80 GB GPU. Median restore took 20.4 seconds against 54.7 for a warm start, over 3 runs each.

Creating that snapshot took 78.5 seconds in one run, so it broke even on the third start. The docs say only dense float16 models at tensor parallelism 1 have been validated. The team uses 4 GPUs per model and local folders, so snapshots do not apply yet.

Frame it as moving slow-to-build state out of a process you restart often. Then follow these steps.

1. 1

   Name the cost

   A restart reads the weights from disk and reruns quantization processing before serving.

2. 2

   Split state from the process

   A daemon per GPU owns the weights. The engine borrows them over CUDA IPC, so it is cheap to replace.

3. 3

   Show the safety net

   A fingerprint covers model files, layout, quantization and version. A mismatch falls back to disk.

4. 4

   Name the blind spot

   The file hash covers shapes, not values, so the deploy process restarts the daemons with every new checkpoint.

5. 5

   State the trade-off

   Zero-copy saves memory but ties the engine to the daemon. Copy mode costs a full copy.

6. 6

   Draw the boundary

   It helps crash restarts and rolling deploys on one machine, not new machines, autoscaling or upgrades.