RoPE Positional Encodings Explained: Token Order and Context Extension

It loses track of where words are, and its answers fall apart. In models that use RoPE positional encodings, the part that encodes word order was trained only up to a fixed length.

Picture a common setup. A team runs an open-weight language model, meaning its trained numbers are published, trained on windows of 2,048 tokens, to answer questions about rental agreements. The setup is made up, but the failure is real and well documented.

Most agreements fit. Then a long agreement with annexures arrives at about 8,000 tokens. The team raises the limit in the serving config, and the answers turn into fluent nonsense.

The paper that introduced position interpolation measured this directly. Pushing such a model past its trained window without any fix gave perplexity above 1,000. Perplexity measures how surprised a model is by real text; lower is better.

The fix is not a bigger server. It is a change to RoPE positional encodings, the mechanism that tells the model where each token sits. To change it safely, you first need to see why it exists.

You need eight words. Each one comes back many times, so read them once now.

• Token: a word or word piece the model reads as one unit. "Agreement" might be one token or two.
• Vector: a list of numbers. The model turns each token into a vector of, say, 4,096 numbers before attention starts.
• Attention: the step where every token looks at other tokens and takes a weighted mix of them. The weights say how relevant each other token is.
• Query, key and value: three vectors made from each token. The query asks, the key advertises, and the value is what gets mixed in.
• Dot product: multiply two vectors number by number and add the results. A large dot product between a query and a key means a high attention weight.
• Context window: the most tokens the model can handle in one request. In our setup it is 2,048 tokens.
• Head dimension: the length of one query or key vector inside one attention head. We use 128, a common size.
• Softmax: the step that turns attention scores into weights that are all positive and add up to 1.

Plain attention compares every query with every key using dot products, and none of that maths looks at where a token sits. Shuffle the input tokens, and each token's output is unchanged; the outputs just move with them.

The original transformer paper says this plainly. Its model has no recurrence and no convolution, so it must be given position information from outside.

Yes, it will. We ran a single attention layer in NumPy 2.5.4 with random weights and five tokens. Shuffling the inputs shuffled the outputs and changed nothing else; the largest difference was 3.3 × 10^-16, which is rounding noise.

For the same word in two different orders, the output vectors matched exactly. Who pays whom is lost. That is why transformers add a positional encoding: extra information that marks where each token sits.

A causal mask leaks some order, but not enough

Text generators use a causal mask, a rule that lets each token see only itself and earlier tokens. That rule breaks the symmetry. In our run the shuffled outputs differed by up to 1.82 once the mask was on.

A 2023 study, arXiv 2305.19466, showed that models with a causal mask and no positional encoding can still learn position. This guide follows explicit schemes, starting with the ones that came before RoPE.

They added a position vector to each token's vector before attention. The two common choices were fixed sine waves and a learned table, and both mix position into the token's content.

Sinusoidal positions: fixed waves

The 2017 transformer paper built each position vector from sine and cosine waves of many speeds. The fastest wave repeats every 2π positions, about 6.3, and the slowest every 10,000 × 2π positions.

The vector is added to the token's vector, not attached at the end, so both must have the same length. The authors hoped smooth waves would let the model handle longer inputs than it trained on.

Learned positions: a lookup table

A learned scheme keeps a table with one trained vector per position. The same paper tried this and found nearly identical results. The catch is the table's size.

A table with 512 rows simply has no row for position 512. In our NumPy run, asking for it raised an IndexError. The model cannot even represent a longer input, let alone understand it.

Why adding a vector is awkward

When position is added before the query and key projections, the attention score mixes content and absolute position. We tested this with sine waves added to two random token vectors.

At the same distance of 3 tokens, the score was 44.71 at positions 5 and 2, 60.12 at 105 and 102, and 42.57 at 1003 and 1000. The model must learn that these all mean "three tokens back".

A 2018 paper, arXiv 1803.02155, moved distance information inside attention itself and reported better translation scores. RoPE takes the same goal and reaches it with a rotation.

RoPE positional encodings rotate each query and key vector by an angle that grows with the token's position. RoPE stands for rotary position embedding, from a 2021 paper, arXiv 2104.09864. Nothing is added; the vector is turned.

Here is the recipe. Split a 128-number query into 64 pairs: numbers 0 and 1, numbers 2 and 3, and so on. Treat each pair as a point on a flat plane, like an arrow from the centre.

Turn pair i by an angle equal to the position times a speed. Counting pairs from 0, the speed of pair i is 10,000 to the power of minus 2i over 128. Do the same to the key, using the key's own position.

The values are not rotated. Only the query and key, the two vectors that are compared, carry position. The paper calls this multiplicative, in contrast to the additive schemes above.

An analogy: a wall of clocks

Think of the 64 pairs as 64 clock hands on a wall, each running at its own fixed speed. A token's position sets the time, and every hand shows that time at its own speed.

The fastest hand moves 1 radian per token and completes a turn every 6.28 tokens. The slowest moves so slowly that one turn takes about 54,410 tokens. Fast hands tell near neighbours apart; slow hands tell far-apart regions apart.

Where the picture breaks: real clock hands carry no content. In RoPE, each hand also has a length and a starting direction that come from the token's meaning. Rotation changes only the direction.

RoPE in 20 lines of NumPy

This is the whole idea in runnable code. Beginners can skip the code; the paragraph after it says what it prints. It rotates the pairs in place using cosines and sines, the fast form the RoPE paper recommends instead of building a big rotation matrix.

import numpy as np

def rope_angles(positions, dim, base=10000.0, scale=1.0):
    # one speed per pair of numbers: base ** (-2i / dim)
    inv_freq = 1.0 / base ** (np.arange(0, dim, 2) / dim)
    pos = np.asarray(positions, dtype=np.float64) / scale  # scale above 1 = position interpolation
    return np.outer(pos, inv_freq)                          # shape (tokens, dim / 2)

def apply_rope(x, positions, base=10000.0, scale=1.0):
    # x has shape (tokens, dim); rotate pairs (x0, x1), (x2, x3), ...
    ang = rope_angles(positions, x.shape[-1], base, scale)
    cos, sin = np.cos(ang), np.sin(ang)
    even, odd = x[:, 0::2], x[:, 1::2]
    out = np.empty_like(x)
    out[:, 0::2] = even * cos - odd * sin
    out[:, 1::2] = even * sin + odd * cos
    return out

rng = np.random.default_rng(1)
q, k = rng.normal(size=(1, 128)), rng.normal(size=(1, 128))
for m, n in [(5, 2), (105, 102), (100003, 100000)]:
    score = (apply_rope(q, [m]) @ apply_rope(k, [n]).T).item()
    print(f"query at {m}, key at {n}: score {score:.10f}")

On NumPy 2.5.4 it prints the same score, -12.7810316744, for all three lines. We also checked that this fast form matches the full block rotation matrix, and that rotation never changes a vector's length.

Because turning both arrows changes the angle between them by exactly the difference of their turns. A query at position m and a key at position n end up separated by the gap, m minus n, times each speed.

The dot product of two arrows depends only on their lengths and the angle between them. So the attention score depends on the content and the distance between the tokens, not on where the pair sits in the text.

This is the property the RoPE paper set out to build. Absolute positions go in, one per token. Relative positions come out, inside every attention score, without any extra parameters.

For the rental agreement, a clause that says "subject to the clause above" now sees the same distance pattern on page 1 and on page 20. The model does not need to relearn it at every location.

Far tokens get a weaker pull, on average

The RoPE paper also shows a long-term decay: the possible size of a score tends to shrink as distance grows. It is a bound and an average trend, not a rule for every pair of tokens.

We lined up 64 arrows of length 1, turned each by distance times its speed, and measured the length of their average. It was 1.00 at distance 0, 0.50 at 64, 0.25 at 1,024 and 0.07 at 4,096. Near tokens line up more easily than distant ones.

Beginners can stop here. You know why order must be added, how RoPE adds it, and why it gives relative distances. The rest is for engineers stretching a context window.

They differ in where position enters and in what happens past the trained length. This table puts the four side by side; ALiBi gets its own section next.

ALiBi adds no position vectors and rotates nothing. It subtracts a fixed penalty from each attention score, proportional to how far back the key is. ALiBi stands for attention with linear biases, from arXiv 2108.12409.

Each head gets its own slope, fixed before training. With 8 heads the slopes are 1/2, 1/4, and so on down to 1/256. We generated them in NumPy and they matched the paper's rule.

A key 1,000 tokens back gets 500 subtracted from its score in the steepest head, and about 3.9 in the gentlest. Steep heads focus on neighbours; gentle heads can still look far.

The paper reports a 1.3-billion-parameter model trained on 1,024 tokens and tested on 2,048. It matched the perplexity of a sinusoidal model trained on 2,048, while training 11% faster with 11% less memory.

In the same paper, a rotary model trained on 1,024 tokens improved only up to about 100 extra tokens. That gap is why RoPE models need the scaling tricks in the next sections.

Some RoPE pairs turn so slowly that, during training, they never completed one turn. At longer lengths they reach angles the model has never seen. The position interpolation paper shows attention scores can then become catastrophically large.

With a head dimension of 128 and the usual speed base of 10,000, our NumPy count showed 41 of the 64 pairs complete a full turn within 2,048 tokens. Each of those pairs has already seen every angle on its own, so longer inputs repeat old angles for it.

The other 23 pairs never finished a turn. At 8,192 tokens, those pairs reach up to 4.0 times the largest angle seen in training. That is our rental agreement at 8,000 tokens.

Fine-tuning, a short extra round of training, helps less than you might hope when done on longer text. The position interpolation paper fine-tuned on longer inputs for over 10,000 steps, and the usable window grew only from 2,048 to 2,560 tokens.

Position interpolation divides every position by a scale factor, so the longest new input maps back inside the trained range. For 8,192 tokens and a 2,048-token model, the factor is 4.

Position 8,191 becomes 2,047.75, a spot the model already knows how to read. In our count, no slow pair went more than 0.04% past its trained angle. The paper, arXiv 2306.15595, derives an upper bound on attention scores about 600 times smaller than extrapolation's, in its test setting.

The paper extended windows up to 32,768 tokens with about 1,000 fine-tuning steps. Without any fine-tuning, the models already showed some language modelling ability at 4 times the length.

The cost: neighbours get crowded

Dividing positions also slows the fast pairs. The fastest pair turned 1 radian per token before and only 0.25 radians after. For that pair, the angle between neighbours is four times smaller, so they are harder to tell apart.

The YaRN paper, arXiv 2309.00071, argues this loses high-frequency detail that tells close tokens apart. It suspects this is why short-text quality dips slightly after interpolation.

They stretch the slow pairs and leave the fast pairs nearly alone. NTK-aware scaling does it by changing the speed base. YaRN does it pair by pair, and also adjusts attention sharpness.

NTK-aware scaling: change the base

NTK-aware scaling first appeared in a public online post and is written up in the YaRN paper. NTK stands for neural tangent kernel, a theory the YaRN paper cites for why networks struggle to learn high-frequency detail. It replaces the base b with b times s to the power d divided by d minus 2. Here s is the scale factor and d the head dimension.

For s of 4 and d of 128, the base goes from 10,000 to about 40,890. In our run the fastest pair kept exactly its old speed, and the slowest pair slowed by exactly 4.

The middle pairs are the weak spot. In our count, 22 of the 23 slow pairs still went more than 1% past their trained angles, reaching up to 1.62 times the trained maximum. The YaRN paper calls these out-of-bound values.

Dynamic scaling: pick the scale per request

Dynamic scaling sets s to the current length divided by the trained length, and never below 1. A 1,000-token request keeps s at 1; a 3,000-token request on our model uses about 1.465.

The YaRN paper reports that dynamic NTK scaling works well with no fine-tuning at all. It also warns that changing s changes every token's rotation, which affects caching.

YaRN: interpolate by parts and cool the softmax

YaRN looks at each pair's wavelength, the number of tokens one full turn takes. It divides the trained length by that wavelength to get a ratio r, then picks a rule per pair.

Pairs that turn more than 32 times in training keep their speed. Pairs that never finish a turn are divided by s, just like position interpolation. Pairs in between get a linear blend. The thresholds 1 and 32 are the paper's recommended values.

In our count with s of 4, no slow pair went more than 0.04% past its trained angle, the same as position interpolation. YaRN also multiplies the query and key by 0.1 times the natural log of s, plus 1. This makes the softmax slightly sharper; the paper fitted the formula to lower perplexity.

The paper reports needing 10 times fewer tokens and 2.5 times fewer training steps than a prior extension method it compares against. Its scale-16 run used 400 steps, and its scale-32 models kept improving out to 128,000 tokens.

Three bugs give wrong answers without an error message: the wrong pair layout, a cache that goes stale under dynamic scaling, and precision drift at large positions. Each one keeps every array shape correct and still changes the scores.

Two layouts for the same idea

Our code pairs neighbours: numbers 0 and 1, 2 and 3. A common alternative pairs number i with number i plus 64, the first half with the second half. Both give relative positions.

They are not interchangeable. In our run, the two layouts rotated the same query into different vectors. Weights trained with one layout and served with the other produce wrong scores, not crashes. vLLM 0.32.0 ships rotate functions for both layouts.

Caching rotated keys

Servers keep a KV cache, the stored keys and values of earlier tokens, so they skip recomputing them. With fixed RoPE, a key's rotation never changes, so caching rotated keys is safe; our check matched exactly.

With dynamic scaling, s changes as the input grows. In our run, keys cached at 2,048 tokens no longer matched keys recomputed at 3,000, with dynamic NTK and with dynamic interpolation alike. The YaRN paper advises caching keys before RoPE is applied when s can change.

Precision drift at large positions

Angles are position times speed, and large positions lose digits in 32-bit floats. With angles in float32, the score that should stay -12.7810 drifted by 0.005 at position 100,003 and 0.033 at 1,000,003.

What the scaling config looks like

In vLLM 0.32.0, as of 27 October 2026, the rope_parameters entry read from the model's config picks the method. The rope_type field takes default, linear, ntk, dynamic or yarn, among others; rope_theta is the base, defaulting to 10000.

In vLLM's source, linear divides positions by factor, which is position interpolation. The yarn type also needs original_max_position_embeddings, the trained length, and defaults its two thresholds to 32 and 1.

Start from the order problem, then show the rotation, then the length limit and its fixes. A strong answer names the trade-off between stretching the window and keeping short-text quality.

1. 1

   State the problem

   Attention is order-blind: shuffled inputs give shuffled outputs. The model needs position information, and it should generalise across where text appears.

2. 2

   Name the old fixes

   Sinusoidal or learned vectors added to token vectors. They mix content with absolute position, and a learned table cannot go past its size.

3. 3

   Explain RoPE

   Rotate each query and key pair by position times a fixed speed. The dot product then depends only on the distance between tokens. Values are untouched.

4. 4

   Explain the length limit

   Slow pairs never finish a turn during training, so longer inputs push them to unseen angles. That is why naive context extension fails.

5. 5

   Pick an extension

   Position interpolation divides positions by the scale. NTK-aware scaling changes the base. YaRN blends per pair and adds a temperature, with short fine-tuning.

6. 6

   Close with operations

   Match the pair layout to the weights, cache keys before RoPE under dynamic scaling, and test long-input retrieval before raising the limit.

For the cost side of long inputs, such as attention's growth with length and KV cache memory, see the long context attention costs guide in the related reads.