Choosing and Upgrading Embedding Models Without Downtime

You can end up with a model that scores higher on public tests but finds worse results for your own users. A leaderboard score is an average over other people's data, and your data may behave differently.

An embedding model turns a piece of text into an embedding, a list of numbers that captures what the text means. Texts with similar meaning get similar lists. Such a list is also called a vector.

Picture a common setup. A small legal-tech team runs case-law search for law firms in Kerala, where a lawyer types a question and the tool finds past court judgments that match. The setup is made up, but the problem is real.

The tool holds about 220,000 judgments, split into 1.8 million chunks, short passages of a few hundred words each. Every chunk got its embedding in 2024, from the model the team picked back then.

Now a new embedding model tops a public leaderboard, about nine points above the team's current model on retrieval. The team's engineer wants to switch that week. The search lead asks two questions first.

Will it find better judgments for these lawyers, or only for the leaderboard? And how do you move 1.8 million vectors without search going dark? The first half of this guide answers the first question, and the second half answers the other.

Embedding model selection is choosing which model turns your documents and queries into vectors, judged by how well it finds the right results for your users. It matters because quality, storage, speed and cost all follow from that one choice.

In a search system, both sides go through the same model. Each chunk becomes a vector once, when you add it, and each query becomes a vector at search time. The system then returns the chunks whose vectors sit closest to the query's vector.

Those vectors live in a vector index, a data structure built to find the nearest vectors quickly. Our guide on vector database architecture covers how that works inside. Here, what matters is that every vector in one index came from one model.

So the model choice touches everything. Longer vectors need more memory, and a slower model makes every search slower. A model that misreads legal language returns the wrong judgments, and no index setting can fix that.

Embedding benchmarks measure how well a model does on a fixed set of public tasks and datasets, then report scores and averages. The best known is MTEB, the Massive Text Embedding Benchmark, which also runs a public leaderboard of models.

The 2022 MTEB paper covered 8 kinds of task across 58 datasets and 112 languages. The tasks include retrieval, clustering and classification. Its authors found that no single embedding method was best across all tasks.

A much larger version, MMTEB, followed in February 2025, with more than 500 tasks in over 250 languages. Its best publicly available model had only about 560 million parameters, the numbers a model learns in training. Models with billions of parameters led on some subsets, so size alone did not decide the winner.

An older benchmark, BEIR, from 2021, tested search models on 18 datasets they were not trained on, which is called zero-shot testing. It found BM25, a classic keyword search method, to be a strong baseline, the simple method to beat. Embedding-based search models often did worse than other approaches on this unfamiliar data.

Benchmarks can also be learned by heart. MMTEB's authors built a smaller zero-shot English version that, among other cuts, leaves out two datasets often used in training. Several smaller English models scored lower on it, and the authors suggest those models had trained on the dropped data.

Here is the problem for the legal search tool. A leaderboard average mixes many tasks it does not care about, like sorting short reviews by mood. Its own task is narrow: Indian legal questions matched against long judgments.

A picture from an attar shop

Think of an attar shop in Kannauj, where perfume oils are sold in small bottles. The label on a bottle says rose, long lasting, fine quality. A careful buyer still dabs it on their own wrist and waits an hour, because the same attar smells different on different skin.

A leaderboard score cannot tell you these things about your own system:

• Your field. Legal terms like res judicata (a matter already decided by a court), or a case citation, may be rare in public datasets.
• Your languages. Lawyers in Kerala often mix English with Malayalam words, and an English-heavy average hides how a model handles that.
• Your document length. Judgments are long, and every model reads only up to a fixed input limit.
• Your costs. A score says nothing about your storage bill, your server speed or your embedding bill.
• Your timing. Leaderboards change as new models are added, so a rank you read today may not hold next month.

You test an embedding model on your own data by building a golden set, running each candidate on it, and counting how often the right documents land near the top. For a few hundred queries, this takes days, not months.

A golden set is a list of real queries, each paired with the documents a person marked as correct. The team takes 400 queries from its search logs. A senior lawyer who advises the team marks the right judgments for each one.

1. 1

   Pick real queries

   Take queries from your logs, not ones you invent. Include short ones, long ones, citations and mixed-language ones, in the share they really appear.

2. 2

   Mark the right answers

   Someone who knows the field marks every judgment that answers each query. Having two people mark the same sample shows how often experts disagree.

3. 3

   Keep a holdout

   Set aside 100 of the 400 queries and do not look at them while comparing. Use them only for the final decision, so your choice does not just fit the test.

4. 4

   Embed the same chunks with each model

   Build one small test index per candidate. Use the same chunks and the same chunk size, so the model is the only thing that changes.

5. 5

   Score, then read

   Compute the scores, then read the queries where the models disagree. That is where you learn why one model is better.

Three scores worth knowing

Hit rate at 10 is the share of queries where at least one right judgment appears in the top 10 results. If 237 of 300 queries get a hit, the hit rate at 10 is 0.79.

Recall at 10 is stricter. For each query, it counts what share of all the right judgments made the top 10, then averages over queries. A query with two right answers and one found scores 0.5, not 1.

MRR, short for mean reciprocal rank, takes 1 divided by the position of the first right answer, and 0 if none appears. It then averages over queries, so a right answer at position 1 counts more than one at position 9.

Pause and predict: the leaderboard leader scored nine points higher on public retrieval. How much better do you expect it to be on the team's own queries? The table shows what the team found.

In the table, ms means milliseconds, and a GPU is a graphics chip used for fast maths. The public score is nDCG@10, a ranking score out of 100, while the team counts hits, so compare which model wins, not the size of each gap. The leader beat the current model by only 3 points, just 9 more queries out of 300. The smaller candidate won by 8 points and costs far less to run.

Reading the disagreements explained why. The leader did well on long, chatty questions. The smaller one did better on terms like anticipatory bail and on queries that mixed English and Malayalam, and most real queries look like that.

Here is the whole selection process in one picture. Start at the leaderboard and follow the arrows to the question box. Notice that the leaderboard only feeds the shortlist and never makes the final call.

Besides accuracy, you need to weigh vector size, speed, cost, input length, languages and where the model can run. A model that wins on quality can still lose on any one of these.

• Dimensions. This is how many numbers each vector holds. More can capture more detail, but they take more memory and make each comparison slower.
• Speed. Every search embeds the query first, so a slow model slows every request. Measure it on your own servers under real load.
• Cost. Add the cost to embed your whole collection once, the cost per query, and the memory the index needs.
• Input limit. Each model reads only up to a fixed number of tokens, the small pieces text is split into. One widely used open-source embedding library cuts longer text off by default without a warning; we read this in its source code, current release as of 19 October 2026.
• Languages. If your users mix languages, your golden set must mix them too.
• Hosting and licence. Law firms may need client queries to stay on your own servers. Check that the licence allows commercial use.

You can check the input limit yourself. Embed a long judgment, then the same judgment with a different last page. If the two vectors are identical, the model never read the end.

Storage maths you can check

Stored uncompressed as 32-bit floats, numbers that take 4 bytes each, 1.8 million chunks at 1,024 dimensions need 7.37 GB of raw vectors, before index overhead. 32-bit floats are the default in the open-source library mentioned above, which can also return 8-bit or binary vectors. Many engines can also compress vectors, so check your engine's sizing guide.

Check it with python3 -c "print(1_800_000 * 1024 * 4 / 1e9)", which prints 7.3728, in decimal gigabytes. Your operating system may show the same bytes as 6.87 GiB, because it counts in units of 1,073,741,824 bytes. At 4,096 dimensions, the raw vectors grow to about 29.5 GB.

Some models are trained so that the first part of each vector works on its own. The idea comes from a 2022 paper, Matryoshka Representation Learning, named after Russian nesting dolls. It reported vectors up to 14 times smaller at the same accuracy, on an image classification test.

If a candidate is trained this way, test the shorter vectors on your golden set too. Keeping 256 of 1,024 numbers would cut the tool's raw vectors to about 1.84 GB. The same paper found that keeping a random part of a normally trained model's vector lost accuracy sharply at small sizes, so measure before you trust a cut.

You cannot mix them because each model learns its own coordinates during training, so a position in one model's vectors means nothing to another. Comparing them still gives numbers, but not useful ones. The exception is a model trained on purpose to match an older one, which a 2020 paper did for image search.

Two models trained separately on similar text still arrive at different number systems. Nothing ties the 17th number of one model to the 17th number of another, even when both produce 768 numbers.

We illustrated this with a simulation of two made-up models whose coordinates are unrelated by design, both 384 numbers long. Matching queries to documents from the same model found the right document 500 times out of 500. Mixing the two models found it 0 times.

Often the sizes differ, and the index refuses the vector. In our test, an open-source nearest-neighbour index built for 768 numbers rejected a 1,024-number vector with a dimensionality error. That failure is loud, which is good.

The dangerous case is when the sizes match. The same index accepted a second 768-number vector without asking which model made it. Search then quietly returns nonsense, because new-model query vectors are compared with old-model document vectors.

In a made-up but common slip during the team's trial, a settings change moved queries to a same-size test model 40 minutes before its index was ready. For those 40 minutes, lawyers got unrelated judgments, and no alarm fired.

Since then, the team follows four rules:

• Store the model name and version with every index.
• Make each search request look up its index once, then take the model from that index's own record, never from a separate setting.
• Keep the original text of every chunk, because you re-embed from text, not from old vectors.
• Put the model version in the key of any cache of query vectors, so old cached vectors are never reused.

You loop over the golden set, compare each query's top results with the marked answers, and average three numbers. The code is short, and writing it yourself shows exactly what each score rewards.

The script below scores one model on three golden-set queries. It needs only Python, and it uses short top-3 lists instead of 10 results to keep the sums easy to follow.

golden = {  # query: the judgments a lawyer marked as right
    "anticipatory bail in a cheating case": {"J104", "J221"},
    "res judicata in a property suit": {"J310"},
    "section 138 cheque bounce limitation": {"J052", "J053"},
}
results = {  # one model's top results for each query, best first
    "anticipatory bail in a cheating case": ["J221", "J007", "J500"],
    "res judicata in a property suit": ["J999", "J400", "J310"],
    "section 138 cheque bounce limitation": ["J600", "J601", "J602"],
}

def score(golden, results, k=10):
    hit = rec = rr = 0.0
    for query, right in golden.items():
        top = results[query][:k]
        found = [d for d in top if d in right]
        hit += 1 if found else 0
        rec += len(found) / len(right)
        rr += next((1 / (i + 1) for i, d in enumerate(top) if d in right), 0)
    n = len(golden)
    return hit / n, rec / n, rr / n

h, r, m = score(golden, results)
print(f"hit@10={h:.2f} recall@10={r:.2f} MRR={m:.2f}")

Run it with python3 and expect exactly this line: hit@10=0.67 recall@10=0.50 MRR=0.44. If you see different numbers, check that you copied the whole script.

You run a paired test on the queries where the two models disagree. If a gap that size could easily come from coin flips, treat the models as tied and decide on cost and speed.

Queries where both models hit, or both miss, say nothing about which is better. Only the split queries matter. In the made-up table, the leader alone hit 27 queries and the current model alone hit 18.

If the two models were truly equal, each split query would be a coin flip. The script below asks how often fair coins give a split at least that lopsided. That probability is called the p-value.

from math import comb

def paired_p_value(a_only, b_only):
    # Exact two-sided sign test on queries where only one model hit.
    n = a_only + b_only
    k = min(a_only, b_only)
    tail = sum(comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * tail)

print(f"leader vs current: p = {paired_p_value(27, 18):.2f}")
print(f"smaller vs current: p = {paired_p_value(39, 15):.4f}")

Run it and expect two lines. The first reads leader vs current: p = 0.23, and the second reads smaller vs current: p = 0.0015.

This is the exact form of McNemar's test, a standard test for paired yes-or-no results. We cross-checked both numbers against the binomial test in SciPy 1.16.2, a Python maths library. It treats each query as independent, so remove near-duplicate queries from the golden set first.

You re-embed without downtime by building a new index next to the old one, filling it in the background, and moving searches over in one step once it is ready. The old index keeps serving the whole time.

This is the same idea as a blue-green deployment, a release method that keeps two copies of production. One copy serves users while you prepare the other. Then you point traffic at the new copy, and the old one waits as a fallback.

The switch uses an alias, a second name that points at an index. Search code always asks for judgments, never for judgments_v1. Elasticsearch, a popular search engine, can remove an alias from one index and add it to another in one request. Its docs call this a single atomic operation: searches see the alias before the change or after it, never in between.

Elasticsearch's aliases documentation shows such a swap and says the alias has no downtime and never points to both targets at the same time. That example uses data streams, Elasticsearch's containers for log-style data. Other search engines and vector databases offer similar switches under different names.

The same page shows one catch on plain indexes. If the remove action fails, for example because the alias is not on that index, the add can still run. The alias could then point at two indexes built by different models. Set must_exist to true on the remove, so the whole request fails instead.

1. 1

   Create the new index

   Make an empty index sized for the new model's vectors, named with a version, like judgments_v2. Record which model it belongs to.

2. 2

   Turn on dual writes

   A dual write means every new or edited judgment is embedded with both models and saved to both indexes. Start this before the backfill, so nothing slips through.

3. 3

   Run the backfill

   A backfill is a background job that re-embeds the old chunks. At 300 chunks a second, all 1.8 million take about 100 minutes (made up speed).

4. 4

   Check the counts

   Compare chunk counts and a random sample of ids between the two indexes. Any gap means the job skipped something.

5. 5

   Run shadow traffic

   Shadow traffic means sending a copy of live queries to the new index while users still see old results. Compare the two result lists and the speed.

6. 6

   Flip the alias

   Move the alias judgments from judgments_v1 to judgments_v2 in one request, with must_exist set to true on the remove. Each search looks up the alias once, then embeds with that index's model and searches that same index. So no query mixes models, even during the flip.

7. 7

   Keep the old index for a while

   Keep dual writes and the old index for two weeks, so rollback stays one change away. Then stop writing to it and delete it.

Here is the tool halfway through. Look at the ingest service, the part that adds new judgments, on one side and the backfill job on the other. Both feed index v2, while lawyers are still served only by index v1.

A backfill that survives crashes and edits

The backfill job must survive crashes, because 1.8 million chunks is a lot to start over. It also must not undo an edit that a dual write saved while the job was running.

The script below is a small, runnable model of that job, with 1,200 chunks, an in-memory index and a stand-in model. It crashes on purpose, then resumes from its checkpoint, a saved note of the last finished id.

import json, os

CHECKPOINT = "backfill_checkpoint.json"
chunks = [{"id": i, "text": f"chunk {i}", "version": 1} for i in range(1, 1201)]
new_index = {}  # stands in for index v2

def embed_many(texts):  # stands in for the new model
    return [[float(len(t))] for t in texts]

def load_checkpoint():
    if os.path.exists(CHECKPOINT):
        with open(CHECKPOINT) as f:
            return json.load(f)["last_id"]
    return 0

def save_checkpoint(last_id):
    with open(CHECKPOINT, "w") as f:
        json.dump({"last_id": last_id}, f)

def upsert(rows):  # write unless the index already holds a newer version
    for chunk, vec in rows:
        old = new_index.get(chunk["id"])
        if old is None or old["version"] <= chunk["version"]:
            new_index[chunk["id"]] = {"version": chunk["version"], "vec": vec}

def backfill(batch_size=500, crash_after=None):
    last_id, done = load_checkpoint(), 0
    while True:
        batch = [c for c in chunks if c["id"] > last_id][:batch_size]
        if not batch:
            return last_id
        if done == crash_after:
            raise RuntimeError(f"crashed after chunk {last_id}")
        upsert(zip(batch, embed_many([c["text"] for c in batch])))
        last_id = batch[-1]["id"]
        save_checkpoint(last_id)
        done += 1

if os.path.exists(CHECKPOINT):
    os.remove(CHECKPOINT)
try:
    backfill(crash_after=2)
except RuntimeError as err:
    print(err)
new_index[1100] = {"version": 2, "vec": [99.0]}  # a dual write of an edit
print("finished at chunk", backfill(), "with", len(new_index), "vectors")
print("chunk 1100 is at version", new_index[1100]["version"])

Run it with python3 in an empty folder and expect three lines: crashed after chunk 1000, finished at chunk 1200 with 1200 vectors, and chunk 1100 is at version 2. If the last line says version 1, the version check in upsert is missing.

Upsert means insert, or overwrite if the id already exists, which makes a repeated batch harmless. Without the version check, the same run ends with chunk 1100 back at version 1, a stale vector nobody notices.

Not every vector store supports a conditional write like this. If yours does not, rerun the backfill over every chunk edited while it ran, after the main pass finishes. Deletes need the same care: keep a tombstone, a marker saying the chunk was deleted, so a stale backfill read cannot bring it back.

You switch when shadow traffic shows the new index is at least as good on real queries and fast enough, and the counts match. Then you flip the alias and keep watching.

Shadow traffic gives the search lead a few signals to read for three days:

• Overlap. How many of the top 10 results match between v1 and v2. Low overlap is not bad by itself, but those queries are the ones to read.
• Speed. The slowest searches matter most, so track p95, the time that 95 out of 100 searches beat.
• Empty or odd results. Any query where v2 returns nothing, or clearly unrelated judgments.
• Golden set again. A rerun of the golden set against the full v2 index, not just the small test index.

After the flip, the signals come from lawyers. Track how often they open a result from the top 3, and how often they retype a query straight away. A sudden rise in retyping is a quiet complaint.

You roll back by pointing the alias back at the old index. This works only if the old index still exists and is still up to date, which is why dual writes continue after the flip.

Without dual writes, every judgment added after the flip would live only in v2. A rollback would then hide new judgments from lawyers, which is a second outage. With them, v1 holds everything, and moving the alias back also moves the query model back.

The diagram shows the decisions from shadow testing to deleting the old index. Follow both No arrows, and notice that neither one leaves lawyers without search.

The whole switch costs some extra money. For about three weeks, the team stores both indexes and embeds every new judgment twice. Using the table's numbers, that is about 12.9 GB of raw vectors instead of 5.5 GB, for a short time.

Explain it in two parts: how you would choose the model with evidence, and how you would switch without users noticing. The switch is a natural follow-up question, so have both parts ready.

1. 1

   Ask about the data

   How many documents, how often they change, which languages, and how fast a search must return. These decide most of the answer.

2. 2

   Describe the evaluation

   A golden set of real queries with marked answers, hit rate or recall at 10 and MRR, a holdout and a paired luck test. Benchmarks only make the shortlist.

3. 3

   Do the cost maths

   Raw storage is chunks times dimensions times 4 bytes for 32-bit floats. Add the time and cost to re-embed everything at a stated speed.

4. 4

   Lay out the migration

   New index beside the old, dual writes, a resumable backfill that never overwrites newer versions, count checks, shadow traffic, then an atomic alias flip with must_exist set on the remove.

5. 5

   Plan the rollback

   Keep dual writes and the old index for a set period, so going back is one change. Delete only after the watch period.

6. 6

   Tie model to index

   Store the model version with the index, read it at query time, and version any query-vector cache.