Contents
Figure 1: Tokenization and Byte-Pair Encoding (BPE) Explained, with a Tokenizer Built from Scratch — the model never sees your text, only the integer sequence your tokenizer hands it
Your language model never reads words. It reads numbers, and a tokenizer decides which numbers. Get that step wrong and the model wastes capacity, chokes on rare words, or charges you triple for non-English text.
This post explains why tokenization exists, how byte-pair encoding works, and then builds a working tokenizer in about 50 lines of Python. Every code block below was executed before publishing, and the outputs are the real ones.
Why Tokenization Exists
Models need a fixed vocabulary of units they can embed. You have three obvious choices, and two of them hurt.
- Whole words: short sequences, a huge vocabulary, and any new word becomes an unknown token.
- Single characters: tiny vocabulary and no unknown words, but sequences get long — a 50-character sentence needs 50 steps of context to represent.
- Subwords: common words stay whole, and rare words split into reusable pieces.
BPE builds that subword middle ground automatically. It starts from bytes, repeatedly merges the most frequent adjacent pair into a new token, and stops when the vocabulary is full. You get a vocabulary far smaller than words and sequences far shorter than characters.
That middle ground is not cosmetic. Sequence length sets your attention cost, vocabulary size sets your embedding table, and the split points decide whether tokenization is one token or five.
Where BPE Comes From
BPE did not start in NLP. Philip Gage published it in 1994 as a data compression algorithm: find the most common adjacent byte pair, replace it with a fresh byte value, and repeat [2].
In 2016, Sennrich, Haddow, and Birch adapted it for neural machine translation of rare words [1]. Their problem was exactly the one above: fixed word vocabularies produced unknown tokens, and character models produced sequences too long to model well.
OpenAI's GPT-2 then popularized a byte-level version for LLMs [3], and most modern models use some variant today. The algorithm is thirty years old and about fifteen lines of code — what changed is the scale it runs at.
How BPE Training Works
Training follows a simple loop:
- Start with bytes. Convert text to UTF-8 bytes, so the initial vocabulary holds exactly 256 entries.
- Count pairs. Scan the data and count every adjacent pair of tokens.
- Merge the winner. Create a new token for the most frequent pair, and replace every occurrence.
- Repeat until you have performed
vocab_size - 256merges.
The output is an ordered merge table, and that table is the tokenizer. Nothing else is needed to reproduce the tokenization — not the corpus, not the counts, just the order of merges.
A Tiny Walkthrough
Take the string aaabdaaabac. Here is what happens:
- The pair
aaappears four times, more than any other. Merge it into a new token, call it Z:Z a b d Z a b a c. - Now
Z aanda beach appear twice. PickZ a, call it Y:Y b d Y b a c. Y bappears twice. Merge it into X:X d X a c.
Eleven symbols shrank to five after three merges. Real tokenizers do the same thing over gigabytes of text with tens of thousands of merges, but the mechanism does not change.
Note the left-to-right, non-overlapping merge rule. In aaa, the first two a characters merge and the third stays alone; the algorithm never merges overlapping pairs. That single rule explains most of the surprising tokens you will see later.
The Code: A Tokenizer From Scratch
Here is a complete byte-level BPE tokenizer. It is short enough to read in one sitting and it runs as written.
from collections import Counter
class BPETokenizer:
"""Minimal byte-level BPE tokenizer."""
def __init__(self):
self.merges = {} # (id, id) -> new id
self.vocab = {i: bytes([i]) for i in range(256)} # id -> bytes
@staticmethod
def _pair_counts(ids):
return Counter(zip(ids, ids[1:]))
@staticmethod
def _merge(ids, pair, new_id):
out, i = [], 0
while i < len(ids):
if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
out.append(new_id)
i += 2
else:
out.append(ids[i])
i += 1
return out
def train(self, text, vocab_size):
ids = list(text.encode("utf-8"))
for new_id in range(256, vocab_size):
counts = self._pair_counts(ids)
if not counts:
break
pair = max(counts, key=counts.get)
ids = self._merge(ids, pair, new_id)
self.merges[pair] = new_id
self.vocab[new_id] = self.vocab[pair[0]] + self.vocab[pair[1]]
def encode(self, text):
ids = list(text.encode("utf-8"))
while len(ids) >= 2:
counts = self._pair_counts(ids)
# apply the pair that was merged earliest during training
pair = min(counts, key=lambda p: self.merges.get(p, float("inf")))
if pair not in self.merges:
break
ids = self._merge(ids, pair, self.merges[pair])
return ids
def decode(self, ids):
return b"".join(self.vocab[i] for i in ids).decode("utf-8", errors="replace")
Three details carry the whole design. vocab starts as 256 single-byte entries so any input is representable. merges maps a pair to a new id, and its insertion order is the training order. And encode picks the pair with the lowest merge rank, not the highest count — more on that below.
Try It
Run it on the toy string from the walkthrough:
t = BPETokenizer()
text = "aaabdaaabac"
t.train(text, 256 + 3) # 256 byte tokens, then 3 merges
print(t.encode(text)) # [258, 100, 258, 97, 99]
print(t.merges) # {(97, 97): 256, (256, 97): 257, (257, 98): 258}
That output matches Karpathy's minbpe project on the same input [4], which is a good sign that the implementation is not accidentally correct. Token 258 is X, 100 is d, 97 is a, and 99 is c. The merge table shows exactly how 97 + 97 became 256, then 256 + 97 became 257, and so on.
Now try something realistic:
corpus = ("low lower lowest newer newest wider widest " * 50) + "hello hello world " * 30
t2 = BPETokenizer()
t2.train(corpus, 300)
s = "lowest newer hello 😉 안녕"
ids = t2.encode(s)
print(len(s.encode("utf-8")), "bytes ->", len(ids), "tokens") # 30 bytes -> 16 tokens
print(t2.decode(ids) == s) # True
The measured output is 30 bytes -> 16 tokens, and the round trip returns the original string exactly. The pieces show what happened:
lowe | st␣ | newe | r␣ | hello␣ | 4 bytes (😉) | ␣ | 6 bytes (안녕)
English words from the training corpus collapse into a few tokens, and the trailing space gets absorbed into them. The emoji and Korean characters, which the tokenizer never saw, fall back to raw bytes — four tokens for one emoji, six for two Korean syllables. Nothing fails, and nothing is lost, which is the point of starting from bytes.
That last line is the practical lesson: 8 characters of Korean cost 6 tokens where hello cost one. Multiply that across a document and you have the non-English pricing gap.
Training vs. Encoding: A Subtle Difference
Here is the point that trips people up: encoding does not re-learn anything. It replays the training-time merges in their original order.
A common misconception says the encoder finds the most frequent pair in your new text and merges it. It does not. Look at encode again: it finds the pair in the current sequence with the lowest merge rank, merges it, and repeats until no known pair remains.
That is why two people with the same merge table always get identical tokens, and why the merge table alone is the whole tokenizer. The frequency counting happened once, during training, and never again.
The failure mode is worth naming. If you merge by frequency at encode time, the same sentence tokenizes differently depending on what else is in the batch, which silently breaks caching, batching, and any evaluation you run.
Why Bytes?
Why start from 256 bytes instead of characters? Because UTF-8 bytes can represent every possible string. GPT-2 and GPT-4 both run BPE on the byte representation of UTF-8 text, so they can encode any input without an unknown-token problem [3][5].
A character-level vocabulary needs thousands of entries to cover Unicode, and it still misses things. Bytes give you complete coverage for 256 ids.
The tradeoff shows up in rare scripts and emoji. Byte tokenization often yields more tokens than word-level tokenization, and text the tokenizer rarely saw costs more tokens per character. That is the biggest practical downside, and it is why tokenizer training data matters as much as model training data.
What Production Tokenizers Add
Our toy skips several details that real systems depend on.
Pre-Tokenization
Most BPE implementations split text on whitespace or a regex pattern first, so merges cannot cross category boundaries. Without it, the cat could merge into one token that spans the space and the next word.
GPT-2 introduced a regex that splits text into chunks — contractions, letter runs, numbers, punctuation, whitespace — and that approach continues in GPT-4 [4]. In the from-scratch code above, pair counts span word boundaries, which is exactly what hurts quality on real data.
Word-Weighted Counting
Efficient implementations count pairs per unique word, weighted by how often each word appears, instead of rescanning the byte stream. Naive counting costs far more time as the corpus grows — the difference between minutes and hours on a real dataset.
Special Tokens
Real vocabularies reserve ids for control signals: end-of-text, padding, chat role markers. The tokenizer must handle them as atomic units, not split them into bytes, and the model learns their meaning during training.
Vocabulary Size
GPT-2 uses 50,257 tokens. Bigger vocabularies compress text better and shorten sequences, but they enlarge the embedding table and leave rare tokens poorly trained. There is no perfect size — it is a tradeoff between sequence length and parameter cost.
Compare Against a Real Tokenizer
Want to see how your toy stacks up? Install tiktoken and compare:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base") # GPT-4 / GPT-3.5 encoding
ids = enc.encode("hello world")
print(ids, [enc.decode([i]) for i in ids])
You will see whole common words become single tokens, with the leading space attached [5]. Run the same string through your 300-token toy and the difference is obvious: production vocabularies are trained on trillions of tokens, not 50 repetitions of a toy corpus.
Hugging Face's tokenizers library gives you a production-quality trainer if you want speed, and Karpathy's minbpe offers a clean reference with a RegexTokenizer you can train on your own data [4].
Quirks That Tokenization Explains
Once you understand tokens, a lot of LLM oddities stop being mysterious:
- Leading spaces matter.
" the"and"the"get different token ids. - Letter counting fails. The model sees tokens, not letters, so "how many r's in strawberry" asks about something it never directly observes.
- Numbers split unpredictably. Long numbers break into chunks of varying length, which makes arithmetic harder.
- Non-English text costs more. Languages underrepresented in the training data need more tokens per sentence, raising both cost and context usage.
- Trailing whitespace changes outputs. An extra space changes the token sequence the model sees.
- Code and indentation behave differently than prose, depending on how the tokenizer handled whitespace.
Ever wondered why your API bill does not match your word count? Now you know.
Common Pitfalls
- Treating tokens as words. A token averages less than a word, and the ratio varies by language.
- Skipping pre-tokenization when you build your own. Merges across word boundaries produce odd tokens.
- Assuming encoding retrains the merges. It replays them in order.
- Training the tokenizer on data unlike your target data. A tokenizer trained on English prose handles code badly.
- Forgetting special tokens. Your model may never see its end-of-text marker correctly.
- Changing the tokenizer after training a model. The embeddings tie to specific token ids, so you would have to start over.
- Merging by frequency at encode time. The same sentence then tokenizes differently depending on its batch.
Wrapping Up
BPE starts from bytes, counts adjacent pairs, merges the most frequent one, and repeats. The ordered merge table becomes your tokenizer, and encoding simply replays those merges. Byte-level starting points guarantee that any text encodes without unknown tokens, and production systems add regex pre-tokenization, special tokens, and speed optimizations on top.
Copy the code above, train it on a text file you care about, and print the learned tokens. Watching your own vocabulary form, one merge at a time, makes the whole idea click — and it makes every token-count surprise afterwards obvious.
What text will you train it on first?
Frequently Asked Questions
What is byte-pair encoding in simple terms?
BPE is a compression trick turned into a tokenizer. Start with raw bytes, count every adjacent pair, merge the most frequent pair into a new symbol, and repeat. The ordered list of merges becomes the tokenizer, and encoding just replays those merges in order.
Why do tokenizers use bytes instead of characters?
UTF-8 bytes can represent every string in existence, so the vocabulary never needs an unknown token. A character-level vocabulary needs thousands of entries to cover Unicode and still misses symbols. The cost is that rare scripts and emoji need more tokens per character.
How many tokens is one word?
There is no fixed ratio. Common English words are often one token, average English text runs about 1.3 tokens per word, and languages underrepresented in the training data, plus code and emoji, need more. That is why API bills do not match word counts.
Does the encoder re-run BPE on new text?
No. Encoding never re-learns merges. It applies the merges learned during training, always choosing the pair with the earliest merge rank, and repeats until no known pair remains. That is why two systems with the same merge table always produce identical tokens.
What vocabulary size should a tokenizer use?
It is a tradeoff. GPT-2 used 50,257 tokens. Bigger vocabularies shorten sequences but enlarge the embedding table and leave rare tokens undertrained. For a small domain corpus, a few thousand merges is usually enough to see the effect.
Sources
[1]Sennrich, Haddow, and Birch, Neural Machine Translation of Rare Words with Subword Units — the 2016 paper that brought BPE to NLP: arxiv.org[2]Byte pair encoding — Gage's 1994 compression algorithm and its later adaptation: wikipedia.org[3]OpenAI, GPT-2 code release — the byte-level BPE implementation that popularized the approach for LLMs: github.com[4]Karpathy, minbpe — minimal reference implementation, including theRegexTokenizerand the toy example reproduced above: github.com[5]OpenAI, tiktoken — the production tokenizer library used forcl100k_baseand GPT-4: github.com