Contextual Retrieval Explained: Chunk Context, Index Cost and Recall

The chunk stops saying what it is about, so search cannot find it. Contextual retrieval is the fix that writes that lost context back. Words like this benefit only make sense next to the heading you cut away.

Picture a common setup. An insurance helpdesk bot answers agents' questions from three policy wordings, for Silver, Gold and Platinum plans, plus one claims guide shared by all three. The setup is made up, but every toy-run score in this guide came from our own test run.

Here is one chunk from the Gold plan, exactly as a splitter, the code that cuts documents into chunks, kept it. This benefit starts after a waiting period of 24 months. It is capped at 50,000 rupees per delivery, for up to two deliveries.

Which plan is that? Which benefit? The heading Maternity cover and the title Gold plan policy wording sat above it, and the splitter left both behind. An agent asks about the maternity cover waiting period on Gold. The chunk shares only the words waiting period with that question, and two other chunks contain them too.

This guide shows how contextual retrieval works, what it did in a small experiment we ran, what it costs and when a cheaper fix is enough.

Eight words carry the whole guide. Each one is tied to the helpdesk bot.

• RAG, retrieval-augmented generation: the bot first searches your documents, then hands the best pieces to a language model that writes the answer.
• Chunk: one small piece of a document, such as a paragraph, stored and searched on its own. The SevyDevy guide on RAG chunking strategies covers where to cut.
• Embedding: a list of numbers that stands for a text's meaning. Texts with similar meaning get nearby numbers, so search can match by meaning.
• BM25: a keyword ranking formula. It scores a chunk higher when it contains the query's words, especially rare words.
• IDF, inverse document frequency: BM25's weight for how rare a word is across all chunks. A word in every chunk is worth almost nothing.
• Index time and query time: index time is when you prepare and store chunks, once. Query time is when a user asks, every time.
• Token: the unit language models read and bill by, roughly a word or part of a word.
• Recall@k: the share of questions whose right chunk appears in the top k results. Recall@1 means it came first.

Contextual retrieval is an indexing step: for each chunk, a model reads the whole document and writes one or two sentences that situate the chunk. You prepend that text, meaning you put it in front of the chunk, then embed the result and index it for BM25.

Nothing changes at query time. The user's question is searched the usual way, but the stored chunks now carry the words and meaning they lost when you split the document.

1. 1

   Split the document

   Cut each document into chunks with whatever splitter you already use.

2. 2

   Ask a model for context

   Send the whole document plus one chunk. Ask for a short line naming the document, the section and what unclear words refer to.

3. 3

   Prepend the line

   Store context line plus chunk as the text to search. Keep the original chunk and the context in separate fields too.

4. 4

   Index both ways

   Embed the enriched text into your vector index, the store that searches embeddings, and add the same text to your BM25 index.

5. 5

   Search as usual

   At query time, run both searches, merge the results and pass the best chunks to the answer model.

An everyday picture helps. Tear one page out of a printed policy book and it still has a running header at the top, such as Gold plan, Maternity cover. The context line is that running header, and the search index is the book's index at the back.

Where the picture breaks: a printed header is exact and free. A model-written line costs money for every chunk, and it can be wrong.

A prompt for the context model can be short. Ours, written for this guide, reads: You will see a document and then one chunk from it. Write one sentence naming the document and section, and what unclear words refer to. Do not add facts.

Because writers rely on what came before. A sentence says it, this benefit or the plan, and trusts the heading and earlier paragraphs to fill in the rest.

Researchers call the repair decontextualization. A 2021 paper defines it as rewriting a sentence to be interpretable out of context while keeping its meaning. Chunks need the same repair, at a larger size.

Policy wordings, manuals and reports make it worse. They repeat the same template in every document, so Silver, Gold and Platinum chunks differ only in a number. Without the plan name, BM25 sees three near-identical chunks.

A 2025 benchmark paper on context in retrieval built test sets on exactly this. Its example is a chunk reading He became emperor in 1804, which cannot match a query about Napoleon without the earlier paragraphs.

The fix is older than the name. The 2020 dense passage retrieval paper split Wikipedia into 100-word passages and prepended each article's title to its passage. That is a context line with no model involved.

It puts the missing words and meaning into the indexed text, so both kinds of search have something to match. BM25 gains the exact words, and the embedding gains the topic.

What BM25 does with the extra words

BM25 adds up a score for each query word found in a chunk. Rare words count more, through IDF. Repeating a word helps only up to a saturation point, set by a parameter called k1.

Long chunks are also scaled down, so they do not win just by containing more words. A parameter called b controls this, and b equal to 1 means full length normalisation.

A context line adds Gold, plan and maternity cover to the Gold maternity chunk. Only Gold chunks contain the word Gold, so it separates them from Silver and Platinum. Maternity then picks the right section.

What the embedding gets from the extra words

An embedding model turns the whole enriched text into one vector. With the context line, the text it reads names maternity cover and the Gold plan, so the vector can reflect them.

We did not run an embedding model, so our own numbers are BM25 only. The papers later in this guide measured embeddings.

A nine-chunk check you can run

This sample uses rank-bm25 0.2.2, the newest release on PyPI as of 29 October 2026. The package does no text cleaning, so we lower-case, split and drop a few common words ourselves. It indexes nine chunks: room rent, co-payment and ambulance for each plan.

from rank_bm25 import BM25Okapi
import re

STOP = {"a", "the", "in", "of", "to", "is", "on"}

def tokens(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]

plans = ["Silver", "Gold", "Platinum"]
chunks = [
    "Room charges are paid up to 1 percent of the sum insured per day.",
    "Room charges are paid up to 2 percent of the sum insured per day.",
    "There is no cap on room charges. Any single private room is paid in full.",
    "The insured person pays 20 percent of every admissible claim.",
    "The insured person pays 10 percent of every admissible claim.",
    "There is no co-payment. The insurer pays every admissible claim in full.",
    "Road transport to hospital is reimbursed up to 2,000 rupees.",
    "Road transport to hospital is reimbursed up to 3,000 rupees.",
    "Road and air transport to hospital are reimbursed up to 2,50,000 rupees a year.",
]
sections = ["room rent limit"] * 3 + ["co-payment"] * 3 + ["ambulance charges"] * 3
contexts = [f"From the {plans[i % 3]} plan policy wording, section on {sections[i]}." for i in range(9)]

query = tokens("room rent limit in the Gold plan")
plain = BM25Okapi([tokens(c) for c in chunks])
contextual = BM25Okapi([tokens(x + " " + c) for x, c in zip(contexts, chunks)])

print("plain:     ", plain.get_scores(query).round(2))
print("contextual:", contextual.get_scores(query).round(2))

Pause and predict: the query names the Gold plan. Before reading on, guess which chunk wins on plain text and which wins with context lines.

Our run printed, with spacing trimmed, plain: [0.59 0.59 0.85 0. 0. 0. 0. 0. 0.] and contextual: [2.17 2.76 2.32 0.15 0.8 0.14 0.15 0.8 0.13]. Each number is one chunk's score, in list order.

What just happened? On plain text, only the word room matched. Silver and Gold tied at 0.59, and Platinum led at 0.85 because its text says room twice.

With context lines, rent and limit matched all three room chunks, and Gold won at 2.76 because only it also matched gold. Platinum came second at 2.32.

Notice that the Gold co-payment and ambulance chunks rose to 0.8 as well. Every Gold chunk now says Gold, so the context line picks the right document. The words of the section still have to pick the right chunk.

It moved the right chunk to first place for 18 of 20 queries, up from 4 on plain chunks. A free header of document title plus section heading got 16 of 20, which was the bigger surprise.

We built 22 chunks: six sections from each of the three plans, plus four from the claims guide. Headings were left out of the chunk text, as a fixed-size splitter often does. Then we wrote 20 questions, each with one right chunk.

We compared four versions of the same chunks. Plain text alone. Document title in front. Title plus section heading in front. Our own context line in front, written as a stand-in for model output.

Mean reciprocal rank averages 1 divided by the right chunk's position. First place scores 1, second scores 0.5, so higher is better.

The order held when we changed the rules. Counting ties in the right chunk's favour, plain text reached 6 of 20 first places and the context line still 18. With no stopword list at all, so words like the and is stayed in, the context line got 17.

What surprised us

• The title alone hurt one query. The Silver maternity chunk fell from first to fourth, because on plain text it was the only chunk containing the word plan.
• Context lifted a rival. For co-payment on the Silver plan, the Platinum chunk won, because it says co-payment in both its text and its context line.
• Synonyms never matched. No chunk or context line contains pregnancy or helicopter. The Gold pregnancy query still came first with context lines, but only through the words Gold, plan and benefit.
• Questions about the shared claims guide did not get worse. The reimbursement chunk scored zero on plain text, and ranked first once the word reimbursement came back through the heading.

What this run cannot tell you

We wrote the chunks, the context lines and the questions ourselves, so the test is small and friendly. We wrote the context lines before the questions, using one rule: name the document and section, resolve unclear words, add nothing else.

A real model may write longer or vaguer lines. Our headings were clean, which is why the free header did so well. In PDFs where headings get lost, the free header has nothing to copy.

Every chunk needs one model call that reads the whole document. So the input tokens per document grow with the number of chunks, not with the document's length alone.

Here is the arithmetic with made-up numbers, which you should replace with your own. A document of 8,000 tokens, 400-token chunks, a 100-token instruction and a 60-token context line.

The cached column assumes cached input tokens cost 10% of the normal price, a made-up rate. Check your provider's real cache price and rules.

Scale it to 10,000 documents. That is 1,700 million input tokens without caching and about 315 million billed-equivalent with it. At a made-up 20 rupees per million input tokens and 80 per million output tokens, that is about 34,960 rupees against 7,258. Both totals include 960 rupees of output.

Why prompt caching fits this job so well

Prompt caching reuses the work a model server already did for an identical start of a prompt. Every call for one document starts with the same instruction and the same document, so only the chunk at the end changes.

Order is everything. Put the instruction first, then the whole document, then the chunk. If the chunk comes first, no two calls share a start and nothing is reused. The SevyDevy guide on prompt caching explains the matching rules.

If you host the model yourself, vLLM's automatic prefix caching does the same job. Its design notes say it hashes each block of cached work by the block's tokens and every token before it, and caches only full blocks. It evicts the least recently used blocks first, so send one document's chunks back to back.

Costs that outlast the first build

• Every edit to a document can change the context of all its chunks. Regenerate them all, or accept slightly stale context lines.
• Your index text grows. In our toy run, chunks went from 12.4 to 21.8 words on average, because the chunks were tiny. Long chunks grow by a smaller share.
• Long documents may not fit in the context model's input. Then you send the section, or a summary, instead of the whole document.

The SevyDevy guide on RAG index freshness covers how to plan for that regeneration. Budget for it before you choose this design.

Papers find that document context helps retrieval and that model-written context costs far more indexing time than cheaper methods. The size of the gain depends heavily on the test set.

nDCG@10 is a ranking score from 0 to 100 here that rewards putting the right chunk near the top of the first ten. The context benchmark paper gives no hardware for its runtime column, so treat the times as relative.

The legal paper also names a failure worth checking in your own system. In a standard pipeline on one contract dataset, over 95% of top results came from the wrong document. Their summary per document roughly halved that rate, at one model call per document instead of one per chunk.

The same legal paper found that a generic summary prompt did no worse than one written with expert guidance. That is one study on English legal text, so test it on yours.

Start with the cheapest fix that matches your failure. Free headers first, then one summary per document, then late chunking or model-written context per chunk.

Late chunking, introduced in a 2024 paper, uses a different idea. A long-context embedding model reads the whole document first. Chunk boundaries are applied afterwards, just before each chunk's token vectors are averaged into one, so each chunk's vector already reflects its neighbours.

Another route rewrites the corpus itself. A 2023 paper indexed propositions, short self-contained facts, instead of passages. That changes your chunking, while contextual retrieval keeps your chunks and adds to them.

Index the enriched text in both BM25 and the vector index, merge the two result lists, then rerank the top few. Context improves what each search finds; fusion and reranking decide the final order.

Reciprocal rank fusion is the usual merge. Each chunk scores 1 divided by k plus its rank in each list, summed over lists. The 2009 paper that introduced it fixed k at 60 in a pilot and never changed it.

That rank-only merge suits contextual retrieval. BM25 scores and vector similarities sit on different scales, and fusion ignores scales entirely. The SevyDevy guide on hybrid search with BM25 and vectors walks through it.

A reranker is a model that reads the query and each candidate together, then reorders the shortlist. Our synonym misses, like pregnancy against maternity, are the kind vector search and rerankers are meant to catch. The SevyDevy guide on reranking with cross-encoders covers that step.

Parent-document retrieval pairs well too. You search small enriched chunks, then hand the answer model the bigger section they came from. The SevyDevy guide on parent-document retrieval covers that hand-off.

Decide what the answer model sees. Our choice is the original chunk plus its context line, since the line tells the model which plan it is reading.

The context line can be wrong, bloated or stale, and each of these quietly hurts search. Treat it as generated data you must check.

• Wrong context: a line that names the wrong plan sends the chunk to the wrong queries. Spot-check a sample of lines against their chunks.
• Boilerplate: words in every context line lose weight. In our run, plan, policy, wording and section appeared in over half the chunks, and rank-bm25 floored their IDF at 0.48.
• Repetition: a line that repeats the chunk's main word can lift that chunk above a better one, as co-payment did in our run.
• Stale lines: an edited document leaves old context on old chunks until you regenerate.
• Hidden changes: store the prompt version and model with each line, so you can tell which lines a prompt change affected.

Measure before and after on your own questions. Build 50 to 100 real questions with their right chunks, then compare recall@k with and without context. The SevyDevy guide on RAG evaluation metrics shows how.

Bring it up when a RAG design has many similar documents, such as plans, product manuals or yearly reports. Show the trade-off, not just the trick.

1. 1

   Name the failure

   Chunks lose their title and heading when split, so similar documents produce look-alike chunks and search returns the wrong one.

2. 2

   Try the free fix

   Prepend document title and section heading from the parsed structure, then measure recall@k.

3. 3

   Add generated context

   For chunks that still fail, a model reads the document and writes one situating line per chunk at index time.

4. 4

   Control the cost

   Order prompts as instruction, document, chunk so prompt caching reuses the document. Process each document's chunks together.

5. 5

   Index and fuse

   Put enriched text in BM25 and the vector index, merge with reciprocal rank fusion, then rerank the top results.

6. 6

   Plan upkeep

   Regenerate context when documents change, version the prompt, and keep a labelled question set to catch regressions.