Contents
Figure 1: Self-Attention Explained Step by Step in PyTorch — the mechanism looks like a diagram and turns out to be six lines of tensor math
Everyone says "attention is all you need," but few people can write the mechanism from memory. The idea looks intimidating on a diagram and surprisingly small in code.
This post builds self-attention one step at a time in PyTorch, explains what every tensor means, and ends with a multi-head module you can drop into a model. Every block was executed on PyTorch 2.15 before publishing, and the outputs shown are the real ones.
The Intuition First
Consider the sentence "The cat sat on the mat." When the model processes "sat," which other words should matter? Probably "cat" and "mat." Attention lets each token look at every other token and decide how much each one matters.
Three vectors make that happen. For every token, the model creates:
- Query (Q): what this token is looking for.
- Key (K): what this token offers to others.
- Value (V): the actual information this token holds.
To update "sat," the model compares its query against every key, turns the similarity scores into weights, and mixes the value vectors using those weights. That is the whole trick. Each token ends up represented in the context of the entire sequence.
Because every token attends to every token — including itself — we call it self-attention. The "self" refers to the fact that Q, K, and V all come from the same input sequence.
One historical note: attention did not start in 2017. Bahdanau and colleagues introduced it in 2015 for machine translation, letting a decoder focus on different parts of the input instead of a single fixed vector [2]. The 2017 transformer paper made it the only mechanism [1].
Step 0: Prepare the Input
Our input X has shape (batch, tokens, embedding_dim). In a real model these vectors come from token embeddings plus positional information, which the original transformer added so the model knows word order. Here we fake it with random numbers:
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
B, T, C = 1, 4, 8 # batch size, number of tokens, embedding dimension
X = torch.randn(B, T, C)
Four tokens, eight numbers each. That is small enough to reason about and big enough that the matrix shapes are real.
Step 1: Project to Queries, Keys, and Values
Three learned linear layers turn each embedding into its Q, K, and V vectors:
d = 8 # size of each query/key/value vector
Wq = nn.Linear(C, d, bias=False)
Wk = nn.Linear(C, d, bias=False)
Wv = nn.Linear(C, d, bias=False)
Q, K, V = Wq(X), Wk(X), Wv(X) # each: (B, T, d)
These matrices start random and get trained, so the model learns what "looking for" and "offering" should mean. The same input X feeds all three projections, which is exactly what "self" means here.
Nothing in this step depends on other tokens. It is a per-token linear map, which is why it parallelises across the whole sequence at once.
Step 2: Compute Scores
Next, compare every query with every key. A dot product measures similarity, and a single matrix multiplication computes all pairs at once:
scores = Q @ K.transpose(-2, -1) # (B, T, T)
Read the result like a table. Row i, column j says how relevant token j looks to token i. A 4-token input gives a 4×4 grid — in the run above, torch.Size([1, 4, 4]).
That T × T matrix is where attention gets its cost. Everything else in the layer is linear in T; this one term is quadratic.
Step 3: Scale the Scores
Dot products grow with vector size, and huge scores push softmax into extremes where gradients vanish. So the transformer divides the scores by the square root of the key dimension:
scores = scores / math.sqrt(d)
That scaling keeps scores from getting too large and stabilizes training gradients. It is a small line with a big effect — forgetting it is one of the most common reasons a hand-written attention layer trains badly.
The reason is variance, not folklore. If the components of q and k are roughly independent with unit variance, their dot product has variance around d, so its standard deviation grows like sqrt(d). Dividing by sqrt(d) puts every dimension back on the same scale.
Step 4: Apply a Causal Mask
Language models generate text left to right, so token 3 must not peek at token 4 during training. A causal mask blocks the future: each token can attend only to itself and earlier tokens.
mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)
scores = scores.masked_fill(mask, float("-inf"))
Why negative infinity? Because softmax turns it into exactly zero weight. torch.triu with diagonal=1 marks everything strictly above the diagonal, which is precisely the future.
Not every transformer uses this mask. Encoder self-attention, as in BERT-style models, lets tokens see the whole sentence. Decoder self-attention uses the causal mask. A third type, cross-attention, lets a decoder attend to encoder outputs. For GPT-style language models, you want the causal version.
Step 5: Softmax Into Weights
Now convert scores to probabilities along each row:
weights = F.softmax(scores, dim=-1) # (B, T, T)
print(weights[0])
Each row sums to 1, and the masked entries become exactly 0. Here is the measured output from the run above:
row sums: tensor([1.0000, 1.0000, 1.0000, 1.0000])
row 0: tensor([1., 0., 0., 0.])
row 3: tensor([0.2440, 0.2743, 0.2231, 0.2586])
Row 0 puts all its weight on token 0, because it cannot see anything else. Row 3 spreads weight across tokens 0 through 3, because the mask allows exactly those four.
Print this matrix on a real sentence and you can watch the model decide where to look. On a trained model, the rows stop looking uniform and start showing structure — pronouns attending to their antecedents, for instance.
Step 6: Take a Weighted Sum of Values
Finally, use the weights to blend the value vectors:
out = weights @ V # (B, T, d)
Each output row is a new vector for that token: a mixture of the values from every token it attended to. The shape matches the input's token axis, so you can stack many of these layers.
This is the only step where information actually moves between tokens. Everything before it computed scores; this is where the mix happens.
Put It Together: A Multi-Head Module
One attention pattern is not enough. Different heads can specialize — one tracking syntax, another tracking nearby words, another tracking rare tokens. In multi-head attention, several heads run in parallel with their own projections, then the model concatenates their outputs and passes them through a final linear layer.
Here is a compact causal multi-head module:
class CausalSelfAttention(nn.Module):
def __init__(self, embed_dim, num_heads, max_len=1024):
super().__init__()
assert embed_dim % num_heads == 0
self.h = num_heads
self.d = embed_dim // num_heads
self.qkv = nn.Linear(embed_dim, 3 * embed_dim, bias=False)
self.proj = nn.Linear(embed_dim, embed_dim)
mask = torch.triu(torch.ones(max_len, max_len, dtype=torch.bool), diagonal=1)
self.register_buffer("mask", mask)
def forward(self, x):
B, T, C = x.shape
q, k, v = self.qkv(x).chunk(3, dim=-1) # each (B, T, C)
# split into heads: (B, T, C) -> (B, h, T, d)
q = q.view(B, T, self.h, self.d).transpose(1, 2)
k = k.view(B, T, self.h, self.d).transpose(1, 2)
v = v.view(B, T, self.h, self.d).transpose(1, 2)
scores = q @ k.transpose(-2, -1) / math.sqrt(self.d) # (B, h, T, T)
scores = scores.masked_fill(self.mask[:T, :T], float("-inf"))
weights = F.softmax(scores, dim=-1)
out = weights @ v # (B, h, T, d)
out = out.transpose(1, 2).contiguous().view(B, T, C) # merge heads
return self.proj(out), weights
attn = CausalSelfAttention(embed_dim=64, num_heads=4)
x = torch.randn(2, 10, 64)
y, w = attn(x)
print(y.shape, w.shape) # torch.Size([2, 10, 64]) torch.Size([2, 4, 10, 10])
That comment is the measured output, not a guess: output (2, 10, 64) and weights (2, 4, 10, 10).
The head-splitting trick deserves a second look. A (B, T, C) tensor reshapes into (B, h, T, d) so each head works on its own slice of the embedding, and C equals h × d. After attention you transpose back and merge the heads.
One fused qkv layer does the same work as three separate projections, only faster, because it is a single matrix multiply instead of three.
Check Your Work Against PyTorch
PyTorch 2.x ships a fused attention function. Compare your manual math against it:
q = torch.randn(2, 4, 10, 16)
k = torch.randn(2, 4, 10, 16)
v = torch.randn(2, 4, 10, 16)
# manual
s = q @ k.transpose(-2, -1) / math.sqrt(16)
m = torch.triu(torch.ones(10, 10, dtype=torch.bool), diagonal=1)
manual = F.softmax(s.masked_fill(m, float("-inf")), dim=-1) @ v
# built in
fused = F.scaled_dot_product_attention(q, k, v, is_causal=True)
print(torch.allclose(manual, fused, atol=1e-5))
On the run above this prints True, with a maximum absolute difference of 4.8e-07. That is float32 rounding, not a logic difference — you have reproduced the core of modern attention [3].
In production, use the built-in version. It dispatches to fused kernels, runs far faster, and uses less memory than the manual sequence of matmuls.
Shape Cheat Sheet
| Tensor | Shape |
|---|---|
Input X |
(B, T, C) |
Q, K, V (single head) |
(B, T, d) |
| Scores and weights | (B, T, T) |
| Output | (B, T, d) |
Multi-head Q, K, V |
(B, h, T, d) |
| Multi-head scores | (B, h, T, T) |
Keep these in mind while debugging, because almost every attention bug is a shape or a dimension index. The T × T score matrix also explains attention's main cost: memory and compute grow with the square of the sequence length, which is why long contexts get expensive [1].
Common Pitfalls
- Forgetting the scale factor. Training gets unstable, often only after a few thousand steps.
- Applying softmax over the wrong dimension. Use the last dimension so each row sums to 1.
- Masking after softmax. Mask scores before softmax, or the rows will not sum to 1.
- Using 0 instead of
-infin the mask. Zero scores still receive positive weight after softmax. - Forgetting positions. Self-attention treats tokens as a set, so add positional information.
- Skipping
.contiguous()before.view(). After a transpose the tensor is not contiguous, andviewraises an error. - Mixing up masks across model types. Encoders should not use a causal mask.
- Building the mask per forward pass. Register it as a buffer once, as the module above does.
Wrapping Up
Self-attention comes down to six steps: project to Q, K, V; compute scores; scale; mask; softmax; take a weighted sum of values. Multi-head attention repeats that in parallel with different projections and merges the results. Once you can write it from memory, transformers stop feeling like a black box [4][5].
Run the module above on random input, then print w[0, 0] and watch one head's attention pattern. Then train it on a tiny text file and look again. Seeing the weights change teaches more than any diagram.
Which head will you inspect first?
Frequently Asked Questions
What is self-attention in one sentence?
Self-attention lets every token in a sequence look at every other token, score how relevant each one is, and blend their value vectors into a new representation. The scores come from a dot product between one token's query and every token's key.
Why divide by the square root of d in attention?
Dot products of d-dimensional random vectors have variance proportional to d, so scores grow as the dimension grows. Large scores push softmax into saturation, where gradients nearly vanish. Dividing by the square root of d keeps the variance roughly constant.
What is the difference between causal and bidirectional attention?
Causal attention masks out future tokens so each position only sees itself and earlier ones, which is required for left-to-right generation. Bidirectional attention, used by encoder models such as BERT, lets every token see the whole sequence. Cross-attention instead attends from a decoder to encoder outputs.
Do I need a mask for self-attention?
You need one for autoregressive language models, otherwise training would let each position peek at the answer it is supposed to predict. Encoders that see the whole input do not need a causal mask, though they still need padding masks for variable-length batches.
Why does attention cost grow quadratically with sequence length?
Because the score matrix is T by T, so both compute and memory scale with the square of the token count. Doubling context length quadruples that cost, which is why long-context inference is expensive and why so much research targets sparse and linear attention.
Sources
[1]Vaswani et al., Attention Is All You Need — the 2017 transformer paper: arxiv.org[2]Bahdanau, Cho, and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate — the 2015 origin of attention: arxiv.org[3]PyTorch,scaled_dot_product_attention— the fused kernel used for the parity check: pytorch.org[4]PyTorch,MultiheadAttention— the reference module withbatch_firstand mask semantics: pytorch.org[5]Harvard NLP, The Annotated Transformer — a line-by-line walkthrough of the original implementation: nlp.seas.harvard.edu