Sam Austinon November 14, 202611 min read

Encoder vs Decoder vs Encoder-Decoder Transformers: BERT, GPT, and T5 Compared

Encoder vs Decoder vs Encoder-Decoder Transformers: BERT, GPT, and T5 Compared
Contents

Figure 1: Encoder vs Decoder vs Encoder-Decoder Transformers: BERT, GPT, and T5 Compared — one 2017 architecture, three different bets about which half to keep

Three famous models — BERT, GPT, and T5 — all grew from the same 2017 transformer paper, yet they behave very differently. One fills in blanks, one writes essays, and one turns French into English.

The difference is not magic. It comes down to which parts of the original transformer each model keeps and how each one trains. Here is how the three families work, how they compare, and how to choose.

One Architecture, Three Ways to Use It

The original transformer had two halves: an encoder that reads the input and a decoder that writes the output. Researchers soon realised you can keep either half, or both.

The core distinction is the attention mask. Encoder blocks apply bidirectional self-attention, where every token can see every other token, while decoder blocks use causal self-attention that restricts each token to previous positions. Here are both masks for a five-token sequence:

import numpy as np
T = 5
bidirectional = np.ones((T, T), dtype=int)       # encoder: everyone sees everyone
causal = np.tril(np.ones((T, T), dtype=int))     # decoder: only the past
print(bidirectional)
print(causal)

The output is a full block of ones for the encoder and a lower-triangular staircase for the decoder:

bidirectional        causal
1 1 1 1 1            1 0 0 0 0
1 1 1 1 1            1 1 0 0 0
1 1 1 1 1            1 1 1 0 0
1 1 1 1 1            1 1 1 1 0
1 1 1 1 1            1 1 1 1 1

That single difference explains almost everything below. A token that sees the future understands its context better, but it cannot be used to generate the future, since generation requires not knowing what comes next. If you want the mechanics of the layer itself, self-attention in PyTorch builds it line by line.

Encoder-Only: BERT and Friends

BERT stands for Bidirectional Encoder Representations from Transformers [1]. It uses a transformer encoder with no causal mask, so every word sees the whole sentence at once.

How It Trains

BERT's main pretraining task is masked language modeling (MLM). The process randomly hides 15% of the tokens and asks the model to predict them from the surrounding context:

the [MASK] brown fox jumps [MASK] the lazy dog   ->  predict: quick, over

The original BERT also trained on next sentence prediction, a binary task asking whether sentence B follows sentence A. Later work showed it was unnecessary — RoBERTa dropped it and improved on BERT's results [5].

Strengths and Limits

Encoders excel at understanding tasks: classification, named entity recognition, and retrieval. Models like RoBERTa and DeBERTa refine the recipe. Encoder embeddings also power semantic search, since each token — and the pooled sentence — gets a rich vector, which is why sentence-transformer search still starts with an encoder.

The catch: BERT cannot generate text. It sees the full input bidirectionally and has no mechanism for producing tokens one at a time. A common beginner mistake is trying to use BERT for open-ended generation — it will produce something, but not something good.

Decoder-Only: GPT and Its Descendants

GPT stands for Generative Pre-trained Transformer, and it keeps only the decoder half [2]. Every token attends only to the tokens before it, and the training objective is simply next-token prediction.

Strengths and Limits

Decoder-only models excel at open-ended generation and few-shot prompting. The family includes GPT-2, GPT-3, and GPT-4, plus open models like Llama and Mistral.

One implementation detail worth noting: these models typically tie the token embedding to the output head, which saves parameters on the largest matrix in the model.

The decoder-only design has won the current LLM era, and the reason is simplicity. One stack, one objective, and it scales cleanly. You can express almost any task as "continue this text", so one model covers translation, summarisation, coding, and chat through prompting alone.

The tradeoff comes from the mask. A decoder-only model uses solely causal self-attention, so earlier tokens never see later ones, which makes it less efficient than a bidirectional encoder for pure representation tasks.

Encoder-Decoder: T5 and BART

The third family keeps both halves. The encoder reads the entire input bidirectionally. The decoder then generates the output autoregressively, and cross-attention layers let each decoder step look back at the encoder's output.

T5: Everything Becomes Text-to-Text

Google's T5 (Text-to-Text Transfer Transformer) made a bold simplification: cast every NLP task as text in, text out [3]. To translate, you feed the model a prefix like translate English to German: That is good. and let it generate the answer.

Translation, summarisation, question answering, and grammatical acceptability all use the identical architecture and training procedure — only the prefix that names the task changes.

Span Corruption

T5 also changed pretraining. Instead of BERT's isolated mask tokens, it uses span corruption: mask 15% of tokens in contiguous spans averaging three tokens long, and replace each span with a sentinel token. The decoder then generates only the missing spans.

Here is a toy version of the same idea:

input:   the <X> fox jumps <Y> lazy dog
target:  <X> quick brown <Y> over the </s>

Note the target ends at the end-of-sequence token, not at an empty sentinel. The decoder emits the masked content in order and stops.

This differs from BERT-style MLM in three ways: it masks contiguous spans rather than isolated tokens, the model generates the masked content instead of predicting it in place, and the target sequence runs much shorter than the full input, which saves compute.

BART

BART takes a related approach with more corruption types — token masking, deletion, infilling, rotation, and permutation [4]. The BART paper describes it as generalising both BERT's masked language model and GPT's autoregressive model. T5 sticks to span corruption only.

What the Controlled Experiments Showed

The T5 paper did not just present a model. It ran controlled comparisons of architectures, objectives, and datasets. The encoder-decoder design won on generation tasks like translation and summarisation, and edged ahead on understanding tasks too, which made it the strongest in that unified framework [3].

One caveat the paper itself invites: the baseline T5 carries about 220 million parameters, roughly double BERT-base's 110 million, because it has both stacks. Part of the advantage comes from extra parameters, not only from architecture — a point worth remembering before you conclude that encoder-decoders are simply better.

Side-by-Side Comparison

Feature Encoder-only Decoder-only Encoder-decoder
Attention Bidirectional Causal Bidirectional encoder, causal decoder, plus cross-attention
Pretraining Masked language modeling Next-token prediction Span corruption or denoising
Strong at Classification, NER, retrieval Open-ended generation, few-shot prompting Translation, summarisation, structured text-to-text
Weak at Generating text Efficient bidirectional representations Added complexity and parameters
Examples BERT, RoBERTa, DeBERTa GPT-2/3/4, Llama, Mistral T5, BART, Flan-T5

Try All Three Yourself

Hugging Face pipelines make the comparison a few lines long. These are the standard small checkpoints for each family:

from transformers import pipeline

# Encoder-only: fill in the blank
fill = pipeline("fill-mask", model="bert-base-uncased")
print(fill("The capital of France is [MASK].")[:2])

# Decoder-only: continue the text
gen = pipeline("text-generation", model="gpt2")
print(gen("The capital of France is", max_new_tokens=10))

# Encoder-decoder: text-to-text with a task prefix
t2t = pipeline("text2text-generation", model="t5-small")
print(t2t("translate English to German: That is good."))

Compare how each one behaves. BERT returns ranked fill-ins for the blank. GPT-2 continues freely and sometimes wanders. T5 follows the task prefix and produces a translation. Seeing the three behaviours side by side teaches more than any table.

The small checkpoints will not impress you on quality, but they show the architectural differences clearly — and they run on a laptop CPU.

How to Choose

  • Classification, tagging, or semantic search: an encoder like BERT, RoBERTa, or DeBERTa gives strong results with small, fast models.
  • Open-ended generation, chat, or code: a decoder-only model.
  • Translation, summarisation, or input-to-output rewriting: an encoder-decoder like T5 or BART fits naturally, and it performs well at modest sizes.
  • Few-shot or zero-shot behaviour from prompting: decoder-only models dominate.
  • Tight latency or cost budget on a narrow task: a small fine-tuned encoder or encoder-decoder often beats a huge general model.
  • Unsure and want flexibility: start decoder-only, because the ecosystem and tooling are richest.

Why Did Decoder-Only Win Chat?

If encoder-decoders performed so well in T5's experiments, why do most large chat models use decoder-only designs?

No single cause is provable, but the practical reasons are visible. Decoder-only models give one pipeline that scales with data and compute. Prompting covers many tasks without separate fine-tuning or task-specific heads. And generation with key-value caching is cheap, because the decoder never re-encodes the prefix.

The tradeoff is real, though: encoder-decoders still win at translation and summarisation per parameter, which is why they remain common in speech and translation systems. Treat the "decoder-only won" story as a market outcome, not a proof of architectural superiority.

Common Pitfalls

  • Using BERT for generation. It cannot generate text autoregressively.
  • Expecting GPT-style models to produce strong sentence embeddings by default. Causal attention limits how well each token sees the whole sentence.
  • Forgetting T5's task prefix. The prefix tells the model what to do.
  • Comparing models of different sizes. Parameter count muddies architecture comparisons.
  • Assuming newer means better for every task. A small encoder often wins on classification cost and speed.
  • Mixing up masking styles. BERT predicts masked tokens in place; T5 generates masked spans.
  • Judging an encoder by its generation output. You are measuring the wrong thing.

Wrapping Up

Encoders read everything at once and shine at understanding. Decoders read left to right and shine at generating. Encoder-decoders combine both through cross-attention and shine at transforming one text into another.

BERT, GPT, and T5 each represent one of those choices, and the right pick depends on your task, your budget, and whether you need generation at all.

Run the three-pipeline snippet on a sentence from your own work and compare outputs. The difference between filling a blank and continuing a sentence is obvious the moment you see it.

Which behaviour surprises you most?

Frequently Asked Questions

What is the main difference between BERT and GPT?

The attention mask. BERT is encoder-only, so every token sees every other token bidirectionally, which makes it strong at understanding tasks like classification and retrieval. GPT is decoder-only, so each token sees only the past, which is what makes autoregressive generation possible.

Can BERT generate text?

Not autoregressively. BERT was trained to fill masked positions from bidirectional context, so it has no mechanism for producing a sequence one token at a time. People sometimes sample from it, but the results are poor compared with a decoder-only model trained for next-token prediction.

What does T5 mean by text-to-text?

Every task is expressed as a string in and a string out. Translation, summarization, and question answering all use the same encoder-decoder and the same training procedure; only the prefix that names the task changes, such as translate English to German.

Why do most chat models use decoder-only architectures?

One stack, one objective, and clean scaling. Next-token prediction turns any task into text continuation, so prompting replaces per-task heads and fine-tuning, and key-value caching makes generation cheap. Encoder-decoder models still win on translation and summarization at small sizes.

Is BERT still useful in 2026?

Yes, for understanding tasks. Fine-tuned encoders remain cheap and fast for classification, named entity recognition, and reranking, and encoder embeddings still power semantic search. They are a poor choice only when the task requires generating text.

Sources

  • [1] Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers — masked language modeling and next sentence prediction: arxiv.org
  • [2] Brown et al., Language Models are Few-Shot Learners — the GPT-3 scaling and in-context learning result: arxiv.org
  • [3] Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — T5, span corruption, and the architecture comparison: arxiv.org
  • [4] Lewis et al., BART: Denoising Sequence-to-Sequence Pre-training — the corruption types and the BERT-plus-GPT framing: arxiv.org
  • [5] Liu et al., RoBERTa: A Robustly Optimized BERT Pretraining Approach — dropping next sentence prediction and the training changes that followed: arxiv.org

What are You Looking For?

esc