Transformers as Associative Memory Explained: How Layers Store and Recall Facts

It gives a confident answer that belongs to a neighbour: the right kind of fact, attached to the wrong item. Transformers as associative memory is one way to see why this happens and where to keep such facts instead.

Picture a common setup. An online shop sells hand block-printed kurtas and bedsheets and lists about 3,000 products. The team fine-tuned a small open model on its catalogue, so a chatbot could answer price questions from what it learned.

Fine-tuning means training an existing model a little more on your own data. The run cost about Rs 60,000. The setup is made up, but the problem is real.

A customer asks the price of kurta K-214. The bot says Rs 1,890, which is the price of kurta K-241. The real price is Rs 1,250. The team runs 500 saved questions through the bot: 41 answers are wrong, and 33 of those mix up near-identical codes.

You need a few ideas. A token is a word or word piece the model reads. A vector is a list of numbers, such as 64 in a row; how many numbers it has is called its dimension or length.

A key is the vector used to find a memory, and a value is the vector handed back. A query is the vector you search with. Similarity scores how alike two vectors are, usually with a dot product: multiply matching numbers and add them up.

Think of a locker room. The key's shape finds the locker, and the value sits inside. Where the picture breaks: a real key opens one locker. Here a query partly opens every locker whose key looks like it, and you get a mix of their contents.

If attention itself is new to you, read the SevyDevy guide on how LLMs work first, then come back.

It means treating each part of a transformer layer as a store of key-value pairs. The store answers a query with a blend of values, weighted by how well the query matches each key.

A Python dictionary needs the exact key. An associative memory is looser: a query that is only close to a key still returns something. But a query close to two keys gets a mix of both values.

Wang's post starts with the simplest store, measures the noise each design lets in, and shows that better stores look like feed-forward layers, then like attention. In a standard transformer, every layer contains attention and a feed-forward part, so each layer holds both kinds of store.

It adds every memory into one grid of numbers, called a matrix. To store a pair, you multiply the value by the key to form a grid, called an outer product, and add that grid to the total.

To read, you multiply the matrix by the query. That works out to a sum of every stored value, each weighted by how similar its key is to the query. The post calls this the outer-product store, and it is the naive first try.

If all keys sat at right angles to each other, recall would be exact. But 64-number keys allow only 64 such directions, so once you store more than 64 memories, keys must overlap.

Wang measures the damage with an interference-to-signal ratio: the size of the leaked noise divided by the size of the correct value. For random keys he estimates it as the square root of one number: memories stored divided by key length.

Capacity is the number of memories a store holds before that ratio reaches 1, meaning the noise is as large as the answer. For the outer-product store with 64-number keys, that happens at about 64 memories.

A smarter linear store, which the post calls the optimal linear store, fits all memories at once with least squares, a best-fit method. It recalls exactly while memories number at most the key length and no key can be built from the others, even for look-alikes. Its capacity doubles to about 128.

Similar keys cause wrong recalls because recall weights every value by key similarity. When two keys nearly match, both values get almost equal weight, and the answer drifts towards the neighbour.

That is one likely story for the shop's bot. Inside the model, "kurta K-214" and "kurta K-241" probably map to very similar vectors, so a query for one pulls hard on both. Nobody has opened this model to prove it, so treat it as a hypothesis.

Here is a toy you can follow by hand. One store holds two prices: Rs 1,250 for K-214 and Rs 1,890 for another item. Real values are vectors; we use one number so the sums stay readable.

The query is K-214's own key, so its similarity is 1. The plain store adds 1 times 1,250 plus the other key's similarity times 1,890. A sharp store, the kind attention uses, multiplies each score by 8, called the sharpness. It then uses exponentials to turn the scores into shares that add up to 1. This step is called softmax.

What just happened? The plain store leaks in every case, since it simply adds weighted values. The sharp store almost removes a clearly different neighbour, but a true look-alike still takes a 31 per cent share.

Use it to decide where each fact lives. Facts that change often or look alike are risky to keep only in trained weights. Put the one fact a question needs into the prompt, where attention can match it.

This is our application of the view, not a claim from the post, which does not discuss retrieval. The team changed the bot in three ways.

• The backend looks up the product code in the catalogue and pastes that one row into the prompt.
• The prompt tells the model to answer prices only from that row, and to say so when no row is found.
• The test set gained 120 look-alike pairs, such as K-214 and K-241, run before every release. An average over random questions hides these collisions.

On the same 500 questions, wrong prices fell from 41 to 2 (made-up numbers). The lookup costs about Rs 4,000 a month. This is retrieval-augmented generation, or RAG: fetching facts at question time and pasting them into the prompt.

If you are new to transformers, this is a good place to stop. The rest opens up the two stores inside a layer, checks the post's numbers, and gives you a script to run.

A feed-forward layer acts as a key-value memory because its two weight matrices behave like a list of keys and a list of values. Geva and colleagues showed this in a paper first posted in December 2020.

The feed-forward layer is the small two-step network in every transformer layer. Its first matrix scores the token against thousands of learned patterns, the keys. An activation function then reshapes those scores, and the second matrix adds up stored vectors, the values, in proportion.

The 2017 transformer used ReLU, which sets negative scores to zero. The post notes that today's language models often use a gated variant called SwiGLU instead.

The paper names the only difference from a classic neural memory: that memory normalises scores with softmax, while the canonical feed-forward layer applies no normalising function.

What the 2020 study found

The team studied a 16-layer language model trained on WikiText-103, a public text dataset. For each key, they collected the training text prefixes that switched it on most, called trigger examples.

• Annotators read the top 25 trigger examples for 160 sampled keys, 10 per layer, and found at least one pattern for every key, 3.6 on average.
• Layers 1 to 9 were dominated by shallow patterns, such as a shared last word. Layers 10 to 16 showed more meaning-based patterns, such as a shared topic.
• From layer 11 up, a value's top predicted word matched the word after the trigger text in up to 3.5 per cent of cases. Random chance was 0.0004 per cent.
• A typical input switched on hundreds of memories per layer, 10 to 50 per cent of the 4,096, and most outputs were a mix of them.

So one answer usually blends many memories, which may help explain why K-214 and K-241 can blur. In 2019, Sukhbaatar and colleagues replaced feed-forward layers with learned memory vectors inside attention and report no loss in performance.

The transformer authors already knew

Version 2 of the 2017 transformer paper, posted on 19 June 2017, had an appendix titled "Two Feed-Forward Layers = Attention over Parameters". It described the feed-forward rows as keys and values.

The post says the passage appears only in versions 2 and 3. We confirmed both and found it missing from versions 4 and 7, the current one, as of 19 October 2026.

That appendix also tested the idea. Swapping the feed-forward layers for attention over learned parameters scored 25.5 and 25.9 BLEU, a translation-quality score, against 25.8 for the base model.

Attention is an associative memory whose keys and values come from the tokens in the prompt, not from trained weights. Each token writes one memory. In a model that writes text one token at a time, each token queries the memories of itself and the tokens before it.

The 2017 transformer paper already used this language, describing attention as mapping a query and a set of key-value pairs to an output. Wang calls attention a non-parametric store, because its memories are not trained weights and the store grows with the context.

As the toy showed, attention scores with softmax, which turns scores into shares that add up to 1. Because softmax uses exponentials, a clearly better match gets a much larger share. A near tie stays close at sharpness 8.

Pause and predict: a random 64-number key scores about 0.125 against a query, and the true key scores 1. Squaring widens that gap from 8 times to 64 times. What does an exponential with sharpness 8 do?

It widens the gap to about 1,100 times: the gap is e to the power 7 (8 times 0.875), about 1,097. That is why weak, accidental matches almost vanish from attention's answer.

For 64-number keys the post uses a sharpness of 8, the square root of 64. Real transformers divide each score by the square root of the key length instead.

The post explains why the two agree. It assumes real queries and keys have length about 8 at 64 numbers, so a matching pair scores about 64, and dividing by 8 leaves 8. The 2017 paper's base model used 64-number keys in each of 8 attention heads.

A Hopfield network is a classic associative memory. In 2020, Ramsauer and colleagues showed that a modern Hopfield network, updated for continuous vectors, recalls with a rule equivalent to transformer attention.

They report it can store exponentially many patterns as vector length grows, and retrieve a well-separated pattern in one update. Exponentially many means the count multiplies with each extra number in the vector.

It depends on the scoring function: the sharper it separates strong matches from weak ones, the more it holds. The post derives capacity formulas for random 64-number keys and checks them by simulation.

We re-ran the same set-up with NumPy 2.5.3: random unit-length keys, random values, the query equal to one stored key, averaged over 6 to 40 trials, with two random seeds. Both runs landed within 10 per cent of the post's figures, but the exact digits move with the seed.

In the formulas, d is the key length, 64 here. The softmax figure is e to the power 14, about 1.2 million. The post puts it at nearly 20,000 times the linear store's 64, and 1.2 million divided by 64 is 18,750.

The post also tests long contexts. With sharpness fixed at 8, its chart shows the ratio at 1.25 for about 2.1 million memories, so noise wins. Raising the sharpness with the logarithm of context length, a trick some long-context models use, drove it down to about 0.004 at 2 million.

Check the numbers yourself

The script below measures the ratio for the outer-product store and a softmax store as the catalogue grows. It needs Python 3 and NumPy; we ran it on Python 3.13 with NumPy 2.5.3. NumPy does not promise identical random numbers across versions, so other versions may differ slightly.

import numpy as np

rng = np.random.default_rng(7)
d = 64                                    # length of each key vector

def noise_to_signal(weights, values):
    signal = weights[0] * values[0]       # the part that is item K-214's price
    noise = values[1:].T @ weights[1:]    # leaked bits of every other item
    return np.linalg.norm(noise) / np.linalg.norm(signal)

for n in (16, 64, 3000, 100_000):
    keys = rng.standard_normal((n, d))
    keys /= np.linalg.norm(keys, axis=1, keepdims=True)   # unit-length keys
    values = rng.standard_normal((n, d))                  # stand-in "price" vectors
    sims = keys @ keys[0]                                 # query = key of K-214
    linear = noise_to_signal(sims, values)                # outer-product store
    soft = noise_to_signal(np.exp(8.0 * sims), values)    # softmax, sharpness 8 (dividing by the sum leaves the ratio unchanged)
    print(f"{n:>7} items  linear {linear:6.2f}  softmax {soft:.3f}")

The code's noise_to_signal is the same interference-to-signal ratio as above. The signal is K-214's own value times its blend weight; the noise is every other value times its blend weight. The ratio has no units: 1.0 means the noise is as large as the answer.

What just happened? The plain store's noise roughly matches the answer at 64 items and is 41 times the answer at 100,000. The softmax store stays at 0.3 or lower, still far below its estimated capacity of 1.2 million.

No. The mechanisms are established research; the step-by-step path to the transformer in this post is Wang's telling.

• Established: attention maps a query and key-value pairs to an output (Vaswani and colleagues, 2017).
• Established: feed-forward layers in a trained model behave like key-value memories (Geva and colleagues, EMNLP 2021).
• Established: attention is equivalent to a modern Hopfield network's update rule (Ramsauer and colleagues, 2020).
• The post's framing: every layer is a memory store, and you could rebuild the transformer by reasoning about better stores.
• The post's estimates: capacity numbers that rest on random keys, which the post itself calls heuristics.

Start with the store, then show the trade-off between weights and prompt.

1. 1

   Define the store

   Each layer can be read as a key-value memory that recalls by similarity, not exact match.

2. 2

   Name the two stores

   Feed-forward layers hold memories in trained weights. Attention holds memories built from the prompt, and that store grows with the context.

3. 3

   Explain interference

   Similar keys leak into each other's answers. Give a pair, such as two product codes with the same digits in a different order.

4. 4

   Compare capacity

   Sharper scoring holds more random memories. Quote the post's numbers as estimates for random keys in one layer.

5. 5

   Make the design call

   Retrieve changing or look-alike facts into the prompt, keep pasted rows distinct, and test near-duplicate items before each release.