Sam Austinon November 14, 202611 min read

Positional Encodings Compared: Sinusoidal, Learned, RoPE, and ALiBi

Positional Encodings Compared: Sinusoidal, Learned, RoPE, and ALiBi
Contents

Figure 1: Positional Encodings Compared: Sinusoidal, Learned, RoPE, and ALiBi — attention sees a bag of tokens, and every positional scheme is a different answer to "where am I?"

Shuffle the words in a sentence and feed them to a bare attention layer, and it produces the same output for each word. Attention has no built-in sense of order. "Dog bites man" and "man bites dog" look identical until you add position information.

How you add it shapes what your model can do, especially at long context lengths. This post compares the four schemes you will meet most often, with runnable snippets for each — all executed before publishing.

Why Position Has to Be Added

Self-attention is permutation invariant by design. Every token attends to every other token through a dot product, and dot products do not care about order. So transformers need an explicit positional signal.

Different encodings make different tradeoffs: parameter count, length extrapolation, compatibility with relative-position reasoning, and implementation complexity. You can group them into three families:

  • Absolute: each position gets a fixed or learned vector added to the embedding.
  • Relative: attention scores get adjusted by a function of the distance between positions.
  • Rotary: position-dependent rotations applied inside the query and key vectors.

Keep one question in mind as we go: what happens when the model sees a sequence longer than anything in training?

Sinusoidal Encodings (The Original)

Vaswani and colleagues added fixed sine and cosine waves of different frequencies to the token embeddings [1]:

PE[pos, 2i]   = sin(pos / 10000^(2i / d_model))
PE[pos, 2i+1] = cos(pos / 10000^(2i / d_model))

Here is a NumPy version, run as written:

import numpy as np

def sinusoidal(max_len, d_model):
    pos = np.arange(max_len)[:, None]
    i = np.arange(0, d_model, 2)[None, :]
    angles = pos / (10000 ** (i / d_model))
    pe = np.zeros((max_len, d_model))
    pe[:, 0::2] = np.sin(angles)
    pe[:, 1::2] = np.cos(angles)
    return pe

pe = sinusoidal(50, 16)

The scheme needs no training and adds no parameters. Low dimensions oscillate fast and high dimensions oscillate slowly, so every position gets a unique fingerprint. Position 0 comes out as [0., 1., 0., 1., ...], exactly as the formula predicts.

The weakness shows up at long lengths. Sinusoidal encodings extrapolate poorly beyond training lengths, and the failure is not graceful — perplexity degrades quickly once you leave the trained range. Today sinusoidal encoding serves mostly as the historical baseline.

Learned Absolute Embeddings

The simplest alternative skips the formula. You allocate a table with one trainable row per position, and add P[pos] to each token embedding. BERT and GPT-2 both used this approach.

The table costs L_max × d_model parameters, which is small next to the rest of the model. Learned embeddings tend to help in shorter contexts, where data is abundant relative to the maximum length and a flexible positional prior beats hand-crafted waves.

The catch: extrapolation fails completely. There are no embeddings for positions beyond the table, and rows for rarely seen positions never trained properly. A model with a 1024-row table simply cannot read a 1025th token without retraining or interpolation tricks.

RoPE: Rotary Position Embeddings

RoPE (Su et al., 2021) took a different approach [2]. Instead of adding a vector to the input, it rotates the query and key vectors by an angle proportional to position. It pairs up dimensions, and each pair rotates at its own frequency.

def rope(x, pos, base=10000.0):
    """Rotate pairs of dims of x (shape (d,)) by angle pos * theta_i."""
    d = x.shape[-1]
    theta = base ** (-np.arange(0, d, 2) / d)
    ang = pos * theta
    x1, x2 = x[0::2], x[1::2]
    out = np.empty_like(x)
    out[0::2] = x1 * np.cos(ang) - x2 * np.sin(ang)
    out[1::2] = x1 * np.sin(ang) + x2 * np.cos(ang)
    return out

Why rotate? Because the dot product of two rotated vectors depends only on how far apart their positions sit. I tested exactly that:

rng = np.random.default_rng(0)
q, k = rng.normal(size=8), rng.normal(size=8)

print(rope(q, 5)   @ rope(k, 2))     # distance 3  -> -1.586475
print(rope(q, 105) @ rope(k, 102))   # distance 3  -> -1.586475
print(rope(q, 5)   @ rope(k, 1))     # distance 4  -> -1.302115

The measured values are -1.586475320662359 twice and -1.302115330496361. Positions 5 and 2 give exactly the same score as positions 105 and 102, because the distance matches. Change the distance and the score changes. That is the relative-position property, built into the attention math with no learned parameters.

One constraint: the head dimension must be even, since RoPE rotates pairs.

RoPE dominates modern open models — Llama, Qwen, Mistral, Mixtral, DeepSeek, and Kimi all use it or a scaled variant.

RoPE's Extrapolation Problem

Naive RoPE does not extrapolate cleanly either. At positions much larger than the training range, the rotation frequencies produce attention patterns the model never saw.

So a whole literature grew around context extension:

  • Position interpolation squeezes new positions into the trained range by rescaling indices [4].
  • NTK-aware scaling adjusts the base frequency instead of the positions.
  • YaRN combines frequency scaling with attention temperature adjustments [5].

The RoPE base is the knob. Raising it stretches the effective wavelength, which is why base=500000 shows up in long-context model configs. These methods work noticeably better than anything available for absolute embeddings, which is a large part of why RoPE won.

ALiBi: Attention With Linear Biases

ALiBi (Press, Smith, and Lewis, 2021) takes the simplest route of all [3]. It adds no position vectors and no rotations. It just subtracts a penalty from the attention score that grows with distance:

def alibi_bias(T, slope):
    i = np.arange(T)[:, None]
    j = np.arange(T)[None, :]
    bias = -slope * (i - j).astype(float)
    bias[j > i] = -np.inf          # causal mask
    return bias

print(alibi_bias(4, 0.5))

The output shows each token penalized by half a point per step back:

[[-0.  -inf -inf -inf]
 [-0.5 -0.  -inf -inf]
 [-1.  -0.5 -0.  -inf]
 [-1.5 -1.  -0.5 -0.  ]]

The diagonal sits at 0, the next-nearest at −0.5, then −1.0, and so on. Each head uses a different slope from a geometric sequence: for 8 heads the slopes run 1/2, 1/4, 1/8, down to 1/256.

The result is a built-in recency bias: nearby tokens matter more, and distant tokens fade smoothly. A model trained at 1024 tokens can often run at 4096 with modest quality loss, which is why BLOOM and MPT adopted it.

The tradeoff is the same bias that makes it work. Strong recency preference helps extrapolation and hurts tasks that need precise long-range lookups.

Side-by-Side Comparison

Method Where position enters Parameters Extrapolation Used by
Sinusoidal Added to input embeddings None Poor Original Transformer
Learned absolute Added to input embeddings L_max × d None without retraining BERT, GPT-2
RoPE Rotates Q and K inside attention None Good, better with scaling Llama, Mistral, Qwen, DeepSeek
ALiBi Bias on attention logits None (fixed slopes) Strong, with recency bias BLOOM, MPT

One line captures the design difference: sinusoidal and learned encodings get added once at the input, RoPE injects position inside the attention dot product, and ALiBi injects it as a bias on the logits.

What the Research Says

Do not take extrapolation claims at face value. A study evaluating encodings on reasoning and math tasks — including ALiBi, Rotary, and absolute encodings — found that the most commonly used methods are not well suited to length generalization on downstream tasks. Perplexity on plain text and success on reasoning at longer lengths can diverge sharply.

Another line of work points the other way for practical use: RoPE-specific extension methods produced significant gains when stretching a pretrained model's window [4][5], and training-free position interpolation can extend context several-fold.

The honest takeaway: nobody has fully solved length generalization. Extension tricks work well enough to ship, but they do not match training at the target length.

How to Choose

  • Training a new LLM from scratch: default to RoPE. The tooling, scaling tricks, and community knowledge all favor it.
  • Need cheap extrapolation with little tuning: consider ALiBi, and accept its recency bias.
  • Small model, short fixed context, plenty of data: learned absolute embeddings still work fine.
  • Teaching or reproducing the original paper: sinusoidal.
  • Fine-tuning an existing model: keep whatever scheme it trained with.
  • Need longer context from an existing RoPE model: try position interpolation, NTK-aware scaling, or YaRN before you retrain.

Switching positional encodings mid-life destroys most of the learned attention patterns, which makes it the single most expensive mistake on this list.

Common Pitfalls

  • Assuming extrapolation means quality. A model that runs at 4× length may still use the extra context poorly.
  • Swapping encodings after pretraining. The model learned its attention patterns around the original scheme.
  • Forgetting that RoPE needs even head dimensions. It rotates pairs.
  • Ignoring the base frequency when extending context. Changing it without adaptation can hurt short-context quality.
  • Treating position as a solved problem. Reasoning tasks still expose length-generalization failures.
  • Confusing absolute and relative. RoPE looks absolute in construction but acts relatively in the attention score.
  • Forgetting the causal mask in ALiBi. The bias is applied to logits, so the mask still has to be there.

Wrapping Up

Attention cannot see order on its own, so every transformer needs a positional scheme. Sinusoidal and learned embeddings add position at the input and extrapolate poorly. RoPE rotates queries and keys so attention depends on relative distance, and it dominates modern LLMs. ALiBi subtracts a distance penalty from attention scores, which extrapolates well and bakes in a recency bias.

Run the snippets above, then change the RoPE base and watch how the scores shift with distance. That five-minute experiment makes the "knob" idea concrete — and it is the same knob that long-context models turn.

Which scheme will you try in your next model?

Frequently Asked Questions

Why do transformers need positional encodings?

Self-attention is permutation invariant: shuffle the tokens and every output stays the same, because the layer only sees a set of vectors. Position information has to be injected explicitly, either by adding a vector to the input, biasing attention scores, or rotating queries and keys.

Is RoPE absolute or relative?

RoPE looks absolute in construction, because it rotates each vector by an angle set by its own position. In the attention score it acts relatively: the dot product of two rotated vectors depends only on the distance between their positions, not on where they sit in the sequence.

Why does RoPE need even head dimensions?

RoPE rotates pairs of dimensions, applying the same angle to each pair, so the head dimension must split cleanly into pairs. An odd head dimension leaves one dimension unrotated, which breaks the relative-distance property.

What is ALiBi and why does it extrapolate well?

ALiBi adds no position vectors at all. It subtracts a penalty from each attention score proportional to the distance between the query and key, with a different slope per head. The recency bias keeps attention local, which transfers to longer sequences better than absolute embeddings do.

Can I change the positional encoding of a pretrained model?

Not usefully. Pretraining shapes the attention patterns around one specific scheme, so swapping it destroys most of what the model learned. You can scale an existing RoPE model with position interpolation or YaRN, but that modifies frequencies rather than replacing the scheme.

Sources

  • [1] Vaswani et al., Attention Is All You Need — the original sinusoidal encoding: arxiv.org
  • [2] Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding — the RoPE paper: arxiv.org
  • [3] Press, Smith, and Lewis, Train Short, Test Long: Attention with Linear Biases — ALiBi: arxiv.org
  • [4] Chen et al., Extending Context Window of Large Language Models via Positional Interpolation: arxiv.org
  • [5] Peng et al., YaRN: Efficient Context Window Extension of Large Language Models: arxiv.org

What are You Looking For?

esc