Multi-LoRA Serving Explained: Batching Many Fine-Tunes on One Base Model

Each full fine-tune is a complete copy of the model's weights, so serving many of them needs many GPUs that mostly sit idle. Most customers send only a few requests a minute, but each copy needs its own memory all day. Multi-LoRA serving avoids this by sharing one base model.

Picture a common setup. An edtech company in Bengaluru runs a doubt-solving assistant for 100 coaching institutes, and each institute wants answers in its own syllabus terms and tone. The setup is made up, but every number comes from our own scripts or a cited source.

The assistant uses a made-up model of about 7.8 billion parameters, the learned numbers inside a model. Stored as 16-bit numbers, the weights take 14.53 GiB. One 80 GB data-centre GPU can hold a few such copies, never 100.

A hundred full fine-tunes would need 1,453 GiB of weights. That is about 20 such GPUs just to hold them, before any memory for running requests. This guide shows how all 100 institutes are served from one GPU instead.

You need seven words. Each one is tied to the coaching assistant.

• Base model: the shared, frozen model that every institute uses. Frozen means its weights stay fixed while adapters are trained.
• Weight matrix: a grid of numbers inside one layer. A layer turns an input vector, a list of numbers for one token, into an output vector by multiplying it by this grid.
• Fine-tune: extra training on one customer's data, so the model answers in their style.
• LoRA adapter: a fine-tune stored as two thin matrices, A and B, per adapted layer. LoRA stands for low-rank adaptation.
• Rank: the thin side of A and B, often 8, 16 or 64. A higher rank can learn more but takes more memory.
• Batch: the set of requests the GPU processes in one step. Bigger batches mean more answers for the same GPU time.
• Tenant: one customer sharing the server, here one coaching institute with its own adapter.

Multi-LoRA serving keeps one copy of the base model on the GPU and applies a different small adapter to each request, even inside the same batch. Each request names its adapter, and the server adds that adapter's effect at every adapted layer.

The idea rests on a 2021 paper on low-rank adaptation, arXiv 2106.09685. It freezes the base weights and trains two small matrices per layer. B is d by r and A is r by k. Here d and k are the layer's output and input sizes, and r is much smaller than both.

For an input x, the layer computes W x + B(A x). W is the frozen weight, and B(A x) is the adapter's side path. The paper scales that side path by a constant alpha divided by r. Only A and B differ between institutes.

An everyday picture helps. Every institute uses the same printed textbook, and each keeps its own thin sheet of margin notes. A student reads the shared page and then the institute's note beside it.

The textbook is the base model, each sheet of notes is an adapter, and the student is one request. The library keeps one textbook and 100 thin sheets, not 100 rewritten textbooks.

Where the picture breaks: margin notes are only read, but an adapter is computed. Its small matrices multiply the token's numbers at every adapted layer, so a larger rank costs real work, not just paper.

Because a merged adapter becomes a full copy of the weights, and then requests for different tenants cannot share one weight matrix. Keeping it separate costs a little compute but lets every tenant share the base.

The maths makes both choices give the same answer. W x + B(A x) equals (W + BA) x, because matrix multiplication is associative and distributes over addition. The script below checks it on random numbers with NumPy 2.5.3.

import numpy as np

rng = np.random.default_rng(0)
d_in, d_out, r = 512, 512, 8        # layer size and adapter rank
alpha = 16                          # scaling numerator, as in the paper
W = rng.standard_normal((d_out, d_in))       # frozen base weight
A = rng.standard_normal((r, d_in)) * 0.01    # adapter "down" matrix
B = rng.standard_normal((d_out, r)) * 0.01   # adapter "up" matrix
x = rng.standard_normal(d_in)                # one token's input vector
s = alpha / r

unmerged = W @ x + s * (B @ (A @ x))   # base path plus small side path
merged = (W + s * (B @ A)) @ x         # one big matrix, adapter baked in

print("max difference:", np.abs(unmerged - merged).max())
print("same result:", np.allclose(unmerged, merged))
print("adapter numbers:", A.size + B.size, "vs base numbers:", W.size)

It prints a maximum difference of about 4.3e-14, which is rounding error, then True. The adapter holds 8,192 numbers against the base layer's 262,144, about 3% here because this test layer is tiny.

The LoRA paper says merging, W = W0 + BA, adds no inference latency, and switching tasks means subtracting one BA and adding another. That suits one tenant at a time, not a mixed batch.

The same paper names the catch. If you merge A and B into W, it is not straightforward to batch inputs for different tasks in one forward pass. Multi-tenant servers such as vLLM therefore keep the side path unmerged.

Each adapted layer stores rank times (input size plus output size) numbers, since A is rank by input and B is output by rank. Multiply by the number of layers and by bytes per number.

Our made-up model has 32 blocks, each with seven linear layers: four in attention and three in the feed-forward part. Token vectors hold 4096 numbers, two attention layers output 1024, and the feed-forward part widens to 14336.

# Made-up model shape: 32 layers, hidden size 4096, 16-bit numbers.
LAYERS, HIDDEN, KV_OUT, MLP, VOCAB, BYTES = 32, 4096, 1024, 14336, 100_000, 2

# (input size, output size) of the seven linear layers in one block
modules = {
    "q": (HIDDEN, HIDDEN), "k": (HIDDEN, KV_OUT), "v": (HIDDEN, KV_OUT),
    "o": (HIDDEN, HIDDEN), "gate": (HIDDEN, MLP), "up": (HIDDEN, MLP),
    "down": (MLP, HIDDEN),
}

def base_bytes():
    per_layer = sum(i * o for i, o in modules.values())
    return (LAYERS * per_layer + 2 * VOCAB * HIDDEN) * BYTES

def adapter_bytes(rank):
    # A is rank x input, B is output x rank, for every adapted layer
    per_layer = sum(rank * (i + o) for i, o in modules.values())
    return LAYERS * per_layer * BYTES

GiB, MiB = 2**30, 2**20
print(f"base weights: {base_bytes() / GiB:.2f} GiB")
for r in (8, 16, 64):
    print(f"rank {r} adapter: {adapter_bytes(r) / MiB:.0f} MiB")
print(f"100 rank-16 adapters: {100 * adapter_bytes(16) / GiB:.2f} GiB")
print(f"100 full fine-tuned copies: {100 * base_bytes() / GiB:.0f} GiB")
# vLLM reserves GPU slots at max_lora_rank, whatever each adapter's rank
for slots, max_rank in ((4, 16), (4, 64), (16, 64)):
    print(f"{slots} GPU slots at max rank {max_rank}: "
          f"{slots * adapter_bytes(max_rank) / GiB:.2f} GiB")

Running it prints 14.53 GiB for the base and 40, 80 and 320 MiB for rank 8, 16 and 64 adapters. MiB means 2 to the power 20 bytes. A hundred rank-16 adapters take 7.81 GiB, against 1,453 GiB for 100 full copies.

Memory grows in a straight line with rank, so doubling the rank doubles the adapter. For the coaching assistant, all 100 rank-16 adapters fit easily in the server's ordinary RAM, the CPU memory beside the GPU.

The GPU only needs the adapters that the current batch uses. The last lines of the script show that cost, which the vLLM section explains.

The server multiplies every token in the batch by the shared base weight in one big matrix multiplication. It then computes each token's side path with that token's own A and B, and adds the two.

The naive way is wrong for this workload. It runs one batch per adapter, so each institute's few requests form a tiny batch. Most steps then carry far fewer requests than the GPU could handle.

Mixing tenants in one batch keeps the big shared multiplication full. Only the thin side path differs per token, and each token finds its matrices through an adapter id, which the server turns into a GPU slot number.

Pause and predict: six tokens share a batch, from three institutes plus one plain base-model request. Will the batched answer differ from running each request alone with a merged weight? Guess before reading on.

import numpy as np

rng = np.random.default_rng(1)
d, max_rank, n_slots = 256, 16, 4    # layer size, slot rank, GPU slots
W = rng.standard_normal((d, d))      # one shared base weight

# Stacked slots, like vLLM's buffers: every slot is max_rank wide.
A_slots = np.zeros((n_slots, max_rank, d))
B_slots = np.zeros((n_slots, d, max_rank))
for slot, rank in enumerate((8, 16, 4, 16)):   # adapters of mixed rank
    A_slots[slot, :rank] = rng.standard_normal((rank, d)) * 0.05
    B_slots[slot, :, :rank] = rng.standard_normal((d, rank)) * 0.05

# Six tokens from different tenants. Slot -1 means "base model only".
ids = np.array([2, 0, -1, 2, 3, 0])
X = rng.standard_normal((len(ids), d))

# 1) Reference: one request at a time with a merged weight.
ref = np.stack([
    X[i] @ (W + (B_slots[s] @ A_slots[s] if s != -1 else 0)).T
    for i, s in enumerate(ids)
])

# 2) Batched: one base matmul, then gather each token's adapter by id.
use = ids != -1
safe = np.where(use, ids, 0)
shrink = (A_slots[safe] @ X[:, :, None])[:, :, 0]         # x A
expand = (B_slots[safe] @ shrink[:, :, None])[:, :, 0]    # (x A) B
batched = X @ W.T + expand * use[:, None]

# 3) Segmented: sort tokens by adapter, one small matmul per segment.
seg = X @ W.T
order = np.argsort(ids, kind="stable")
for s in np.unique(ids[use]):
    rows = order[ids[order] == s]
    seg[rows] += (X[rows] @ A_slots[s].T) @ B_slots[s].T

print("batched matches per-request:", np.allclose(batched, ref))
print("segmented matches per-request:", np.allclose(seg, ref))
print("segments (adapter: tokens):",
      {int(s): int((ids == s).sum()) for s in np.unique(ids)})

It prints True twice and the segments -1: 1, 0: 2, 2: 2, 3: 1. Both batched methods match the one-at-a-time merged results, including the token with no adapter.

What just happened? The gather method picks each token's A and B by its id, then multiplies. The segmented method sorts tokens by adapter and runs one small multiplication per group, so each adapter's matrices are fetched once per group, not once per token.

Gathering is simple, but it reads a separate copy of an adapter for every token. Grouping the same adapter's tokens together saves that repeated reading, which matters on a GPU where memory traffic often limits speed.

A 2023 paper on multi-tenant LoRA serving, arXiv 2310.18547, built a GPU kernel, a function that runs on the GPU, for this. It calls it segmented gather matrix-vector multiplication, SGMV, and groups inputs for the same adapter next to each other.

The paper names the two steps shrink, from full width down to the rank, and expand, from the rank back up. It keeps a single copy of the base model per GPU.

It reports 12 times the throughput, meaning answers per second, of the serving systems it compared against. It adds about 2 ms of waiting time per generated token. Those are 2023 results on its own test setup, not a promise for yours.

vLLM 0.32.0 follows the same pattern. We read in its 0.32.0 source that the main GPU kernel path sorts the batch's tokens by slot number before running its shrink and expand kernels. When no token uses an adapter, the LoRA kernels are skipped.

vLLM keeps adapters in three places: on disk, in a CPU-memory cache and in a fixed number of GPU slots. A request's adapter moves up these tiers when needed, and the least recently used one is pushed out.

You start the server with --enable-lora and list adapters with --lora-modules, written as name=path. A client then picks an adapter by putting its name in the request's model field. The /v1/models endpoint lists each adapter with its base model as parent.

Here is the path we read in the 0.32.0 source. The worker, the process that runs the model on a GPU, loads a new adapter from disk into CPU memory first. It checks the adapter's settings before evicting anything from the CPU cache.

The CPU cache holds up to --max-cpu-loras adapters. The GPU holds up to --max-loras adapters in slots, and making an adapter active copies its weights into a free slot. Both levels evict the least recently used adapter when full.

Adding adapters while the server runs

You can add adapters without a restart, but the docs mark it as a security risk. Set the environment variable VLLM_ALLOW_RUNTIME_LORA_UPDATING to True, then post the name and path to /v1/load_lora_adapter.

The docs say this should not be used in production unless the environment is isolated and fully trusted. Anyone who can call the endpoint can make the server load an adapter from any path they choose.

A matching /v1/unload_lora_adapter removes an adapter. Setting load_inplace to true replaces an adapter's weights under the same name. Resolver plugins, small add-ons that find an adapter by its name, can also fetch an unknown adapter on first use. They need the same environment variable.

The built-in resolvers read from a local folder or download from a public model hub, and the docs call remote downloads insecure for production. You can also write your own resolver.

Four flags matter most: --enable-lora, --max-loras, --max-lora-rank and --max-cpu-loras. We read every default from the vLLM 0.32.0 source, mainly vllm/config/lora.py, and checked the LoRA docs page for the same version.

With the defaults, only one adapter can be in any batch. For the coaching assistant, that means requests from different institutes take turns, which defeats the point. A starting point might be --max-loras 8 and --max-cpu-loras 100. In our made-up model, 8 rank-16 slots take 0.625 GiB of GPU memory.

What max-loras does inside the scheduler

At each step, the 0.32.0 scheduler first collects the adapter ids of requests already running. It then admits waiting requests one by one.

A waiting request is skipped when the batch already has max-loras distinct adapters and its adapter is not one of them. Requests for adapters already in the batch, and plain base-model requests, can still join.

A skipped request goes back to the front of the waiting queue for the next step. This behaviour comes from vllm/v1/core/sched/scheduler.py and is undocumented as of 28 October 2026; pin your version.

Why max-lora-rank costs memory even for small adapters

vLLM sets aside the GPU slots as fixed blocks of zeros when the model loads, each sized for --max-lora-rank. A smaller adapter is copied into the top-left corner of its slot, and the rest stays zero.

# A rank-8 adapter stored zero-padded in a rank-64 slot gives the same output,
# but the slot costs rank-64 memory and the kernel does rank-64 work.
import numpy as np
rng = np.random.default_rng(2)
d, r, R = 128, 8, 64
A = rng.standard_normal((r, d)); B = rng.standard_normal((d, r)); x = rng.standard_normal(d)
A_pad = np.zeros((R, d)); A_pad[:r] = A
B_pad = np.zeros((d, R)); B_pad[:, :r] = B
print("padded equals unpadded:", np.allclose(B @ (A @ x), B_pad @ (A_pad @ x)))
print("numbers stored, rank 8 vs rank-64 slot:", A.size + B.size, A_pad.size + B_pad.size)

It prints True, then 2048 and 16384. The padded adapter gives the same answer, but its slot stores eight times the numbers, and the kernels work at the slot's full width.

In our made-up model, 4 slots at rank 16 take 0.31 GiB, while 4 slots at rank 64 take 1.25 GiB. Sixteen slots at rank 64 take 5 GiB, counting only the seven linear layers per block.

The slots are created before vLLM measures the memory left for the KV cache, the saved keys and values of earlier tokens. So oversized slots quietly shrink the KV cache, and with it the batch size.

The vLLM docs give the same advice: set --max-lora-rank to the highest rank among your adapters, not far above it. An adapter whose rank exceeds the setting is rejected at load with an error naming both ranks.

Check three things: the adapter is listed, requests really use it, and requests are not stuck waiting for a slot. These steps use vLLM 0.32.0's own endpoints and metrics.

1. 1

   List the models

   Call /v1/models on the server. Each adapter should appear by its name, with the base model shown as its parent.

2. 2

   Compare one prompt

   Send the same prompt with the model field set to the base model and then to an adapter. Use temperature 0, which picks the most likely next token each time, so replies are mostly repeatable. Run each prompt twice. The replies should differ wherever the adapter learned something visible.

3. 3

   Read the LoRA gauge

   The metric vllm:lora_requests_info, on the server's /metrics endpoint for the Prometheus monitoring system, has labels running_lora_adapters, waiting_lora_adapters and max_lora. In 0.32.0 the first two hold adapter names only, as comma-separated text. An adapter with any request in flight appears in both.

4. 4

   Watch for slot pressure

   Watch vllm:num_requests_waiting_by_reason with reason deferred. As of 0.32.0 it counts requests held back by the adapter limit, KV transfer or a blocked status. If it stays above zero while few requests run, raise --max-loras.

5. 5

   Recheck the KV cache

   After raising --max-loras, read the KV cache size in the startup log again. Each extra slot comes out of that memory, so the batch may shrink.

6. 6

   Time a cold adapter

   Adapters listed with --lora-modules load into the CPU cache at startup. Time a request for an adapter evicted since then against a warm one; the gap is your load cost.

Three things change: caches must stay separate per adapter, updated adapters need new paths, and adapter files need checking like any other input.

• Prefix caching, which reuses saved keys and values for a repeated prompt start, includes the adapter's name and path in its block hash, the fingerprint of each cached block. Two institutes with the same system prompt, the fixed instructions placed before every chat, but different adapters never share those blocks.
• That separation is correct, because an adapter changes the numbers from which keys and values are computed. It also means a shared system prompt is cached once per adapter.
• Reloading changed weights in place from the same path through /v1/load_lora_adapter logs a warning in 0.32.0: cached blocks from the old weights are not cleared. Load each new version from a new path.
• Check rank and format before accepting a tenant's adapter. vLLM 0.32.0 rejects adapters with bias terms, adapters using DoRA, a variant that splits weights into size and direction, and any rank above --max-lora-rank.

The block-hash and reload-warning details come from the 0.32.0 source and are undocumented as of 28 October 2026. The SevyDevy guide on PagedAttention explains how those KV blocks are stored and shared.

When multi-LoRA serving is the wrong choice

• One tenant sends almost all traffic. Merge its adapter into the weights and serve it alone, which removes the side path.
• Tenants use different base models. Adapters only share a server when they were trained on the same base.
• Adapters are huge. At high rank on every layer, adapters stop being small, and full fine-tunes or separate servers may be simpler.

They push the same idea to thousands of adapters by managing adapter memory as carefully as the KV cache. Two 2023 papers show the main moves.

A paper on serving thousands of concurrent LoRA adapters, arXiv 2311.03285, stores every adapter in main memory and fetches only those used by running requests to the GPU. It prefetches adapters for waiting requests while the current batch runs.

It also puts adapter weights of different ranks and the KV cache into one paged memory pool, which reduces fragmentation, memory that is free but unusable. It reports serving 2,000 adapters on one GPU.

Against vLLM with naive LoRA support at the time, it reports up to 4 times the throughput. vLLM has changed a lot since 2023, so treat that comparison as history, not a current benchmark.

The multi-tenant paper from earlier, arXiv 2310.18547, adds a cluster scheduler. It sends a new request to the GPU already running the most requests, its working set, and moves requests to other GPUs when one runs out of KV cache space.

Start from the cost of full fine-tunes, show the memory maths, then explain mixed batches and the cache tiers. Interviewers want numbers and trade-offs, not paper names.

1. 1

   Clarify the load

   Ask how many tenants, how much traffic each sends, the base model size and the adapter ranks. Here, 100 institutes, light traffic each, one 14.53 GiB base.

2. 2

   Do the memory maths

   Full copies need 1,453 GiB. Rank-16 adapters need 80 MiB each, 7.81 GiB for all 100, plus one base.

3. 3

   Keep adapters unmerged

   Explain W x + B(A x) equals (W + BA) x. Merged weights are a full matrix per tenant, so tenants could not share one multiplication.

4. 4

   Batch across tenants

   One base multiplication for all tokens, then shrink and expand per adapter, grouped by adapter id so each adapter is read once.

5. 5

   Design the tiers

   Object storage or disk, a CPU cache with least-recently-used eviction, then a few GPU slots. Map them to --max-cpu-loras and --max-loras.

6. 6

   Name the limits

   Slots are sized for the max rank and come out of KV cache memory. Too few slots make tenants wait; too many shrink the batch.

7. 7

   Close with routing and metrics

   Route each tenant to replicas that already hold its adapter, one option to cut cold loads. Watch deferred requests, cold-load latency and KV cache use.