Sam Austinon November 14, 202612 min read

LayerNorm vs RMSNorm and Pre-Norm vs Post-Norm: What Changed in Modern LLMs

LayerNorm vs RMSNorm and Pre-Norm vs Post-Norm: What Changed in Modern LLMs
Contents

Figure 1: LayerNorm vs RMSNorm and Pre-Norm vs Post-Norm: What Changed in Modern LLMs — two quiet edits to the 2017 block that made hundred-layer stacks trainable

Open the code for Llama, Mistral, or Qwen, and you will spot two quiet changes from the original 2017 transformer. The normalization layer got simpler, and it moved to a different spot in the block. Neither change grabs headlines, yet together they make deep models train reliably.

Here is what each change does, how it looks in code, and where the tradeoffs still bite.

What Normalization Does

Deep networks drift. Activations grow or shrink as they pass through layers, which makes training erratic. Normalization layers rescale activations at each step to keep them in a sane range. Without it, magnitudes drift across layers, softmaxes saturate, and training breaks.

Two separate design choices matter here:

  1. Which normalization: LayerNorm or RMSNorm.
  2. Where you place it: before the sublayer (pre-norm) or after the residual addition (post-norm).

These choices are orthogonal. You can mix any normalization with any placement, and the history of the field is exactly that — four combinations tried in different orders. Let us take them one at a time.

LayerNorm: The Original

LayerNorm normalizes along the feature dimension, independently for each token and independent of the batch. It subtracts the mean, divides by the standard deviation, then applies a learned scale and shift. That means two reductions (mean and variance) and two learned parameter vectors.

def layernorm(x, g=1.0, b=0.0, eps=1e-5):
    mu = x.mean(-1, keepdims=True)
    var = x.var(-1, keepdims=True)
    return g * (x - mu) / np.sqrt(var + eps) + b

It quickly became the standard for transformer architectures, and the original transformer used it.

The eps term is not decoration. Without it, a token whose activations are all near zero divides by zero and produces infinities that poison the whole batch.

RMSNorm: The Simplification

Zhang and Sennrich proposed RMSNorm in 2019 and asked a sharp question: do we actually need the mean subtraction? Their answer was no. RMSNorm drops mean-centering and divides by the root mean square only, then applies a single learned gain. It has no mean computation and no shift parameter.

def rmsnorm(x, g=1.0, eps=1e-5):
    return g * x / np.sqrt((x**2).mean(-1, keepdims=True) + eps)

Does skipping the centering hurt? I compared both on random 64-dimensional vectors:

zero-mean input   max|LN - RMS| = 2.8e-17
input with mean +3   max|LN - RMS| = 2.2827

When the input has zero mean, the two layers are numerically identical — the 2.8e-17 is float rounding, not a difference. With a large mean offset they diverge, because LayerNorm removes the offset and RMSNorm keeps it.

The research claim is that models learn to cope. The original paper reported RMSNorm matching LayerNorm in translation quality across WMT benchmarks, and Llama, Gemma, Mistral, and Qwen later confirmed the finding at scale [1].

RMSNorm Is Not L2 Normalization

One clarification worth having straight: RMSNorm is not the same as dividing by the L2 norm. It equals L2 normalization scaled by sqrt(d), which keeps typical outputs near unit scale per element rather than unit norm.

I checked that numerically too: max|rmsnorm(x) - x/||x|| * sqrt(d)| = 0.0, exactly zero with eps=0. The two differ only by a constant factor per vector, which matters for interpretability, not for training.

Why Labs Switched

The payoff is speed and simplicity. Reported speedups run from roughly 10 to 20 percent depending on the kernel and hardware, so treat any exact number as implementation-dependent. At LLM scale even modest savings matter, since normalization runs twice per block across every layer — a 70-layer model calls it 140 times per forward pass.

The adoption list reads like a who's who: Llama, Mistral, Qwen, DeepSeek, and Gemma all use RMSNorm.

Pre-Norm vs Post-Norm

Now placement. Here are the two layouts:

Post-norm (original):   x_next = Norm(x + Block(x))
Pre-norm  (modern):     x_next = x + Block(Norm(x))

In post-norm, you add the sublayer's output to the residual, then normalize the sum. In pre-norm, you normalize the input to the sublayer, and the residual path passes through untouched.

Why Post-Norm Struggles

Post-norm puts a normalization layer on the main highway. Every gradient flowing backward must squeeze through a normalization step at every layer. The original design preserves strong representational fidelity but tends toward training instability in deep models, and post-norm stacks generally need learning-rate warmup to survive.

The commonly cited threshold — instability past roughly a dozen layers without warmup tricks — is a rule of thumb, not a law. It varies with initialization, width, and learning rate.

Why Pre-Norm Won

Pre-norm leaves the residual stream clean. Gradients flow through an unobstructed identity path, which greatly improves gradient flow and training stability. That is why it became the de facto standard for large-scale language models, and why modern stacks with 24 or more layers train at all.

This is the single most important reason today's deep LLMs train without elaborate learning-rate gymnastics. The tradeoff moved from "can it converge" to "how do we control scale", which is a much easier problem.

Pre-Norm's Catch

Pre-norm is not free. Because the main path never gets re-normalized, hidden-state magnitudes tend to accumulate as depth grows. I simulated this with random sublayers of unit-ish gain:

depth   pre-norm rms(x)   post-norm rms(x)
  4          3.04              1.00
 16          5.90              1.00
 64         10.56              1.00

In this toy setup the pre-norm residual stream grows steadily with depth, tracking roughly the square root of the layer count (3.04 → 5.90 → 10.56 against sqrt(4)=2, sqrt(16)=4, sqrt(64)=8), while post-norm holds the scale fixed at 1.0.

A trained model behaves differently from random weights, so do not over-read these numbers. Researchers note that variance growth at initialization does not persist as training progresses, yet large activation spikes can still persist across layers through residual connections [3].

Gradient problems can still occur too. Xiong et al. observed that pre-LN has more stable gradients than post-LN, but can still hit gradient spikes and divergence in deep models or large-scale pretraining [2].

A Third Option: Normalizing Around the Sublayer

Because both classic placements have flaws, newer models quietly try a hybrid. The Peri-LN paper describes a third strategy that several open-source models adopted with little explanation: applying normalization around the sublayer, on both its input and output [4].

Other proposals combine norms at different points of the residual path, such as Peri-Norm and Hybrid-Norm, to merge the benefits of both paradigms.

Many recent models also normalize queries and keys inside attention (QK-norm) to tame logit growth — Gemma 2 and several Qwen variants do this. Check each model's technical report for the exact placement, because the details differ per family.

What a Modern Block Looks Like

Here is the recipe in code. It is my own PyTorch sketch, tested on PyTorch 2.15:

import torch
import torch.nn as nn

class RMSNorm(nn.Module):
    def __init__(self, dim, eps=1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))     # single learned gain

    def forward(self, x):
        inv_rms = torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
        return x * inv_rms * self.weight

class Block(nn.Module):
    def __init__(self, dim, attn, ffn):
        super().__init__()
        self.norm1, self.attn = RMSNorm(dim), attn
        self.norm2, self.ffn = RMSNorm(dim), ffn

    def forward(self, x):
        x = x + self.attn(self.norm1(x))    # RMSNorm -> attention -> add
        x = x + self.ffn(self.norm2(x))     # RMSNorm -> FFN -> add
        return x

That matches the standard modern block: RMSNorm, multi-head attention, add to the residual stream, then RMSNorm, a gated feed-forward layer, and add again. Two normalizations per block, one before each sublayer.

Remember the final norm after the last block and before the output head, because pre-norm leaves the stream un-normalized at the end.

PyTorch 2.4 and later ship a built-in nn.RMSNorm, so check your installed version before depending on it. The hand-written version above is worth keeping anyway, because the whole point is how little there is to it.

How to Choose

  • Training a new decoder-only LLM: RMSNorm with pre-norm. There is no compelling reason to pick LayerNorm for new autoregressive training.
  • Reproducing an older model such as BERT or GPT-2: match the original — LayerNorm, and match its placement too.
  • Fine-tuning an existing model: keep its normalization exactly as trained.
  • Training very deep models and seeing instability: look at extra stabilizers like QK-norm, or Peri-LN style sandwich normalization.
  • Studying small models: try both and compare, since the differences stay small at small scale.

Common Pitfalls

  • Swapping norms in a pretrained model. The learned weights assume the original normalization.
  • Forgetting the final norm in a pre-norm stack. Logits then inherit whatever scale the depth produced.
  • Mixing up the formulas. Post-norm normalizes the sum; pre-norm normalizes only the sublayer input.
  • Assuming RMSNorm always wins on speed. Gains depend on kernels and hardware.
  • Treating pre-norm as risk-free. Activation growth and spikes still happen at scale.
  • Skipping epsilon. Without it, near-zero activations produce infinities.
  • Applying the gain before the normalization. The gain is a per-channel scale applied after the reduction, not a pre-scaling.

Best Book to Read

CoverBookDescriptionGet it
Cover of “Build a Large Language Model (From Scratch)” Build a Large Language Model (From Scratch)by Sebastian Raschka builds a GPT-style model line by line, including the normalization and residual details this post compares. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Read the chapter on transformer blocks after you have read this post, because Raschka implements the pre-norm block in full and you can see the two norm calls in their real context. It is the shortest path from "I understand the diagram" to "I have a working model."

Wrapping Up

RMSNorm keeps LayerNorm's rescaling and drops its mean-centering and bias, which cuts cost with little or no quality loss. Pre-norm keeps the residual stream clean, which makes deep stacks trainable. Together they define the modern LLM block, though researchers still explore hybrids to fix pre-norm's activation growth.

Run the toy simulations above, then change the depth and watch the residual growth curve. A few lines of NumPy show the pre-norm tradeoff better than any diagram.

Which placement would you pick for a 100-layer model?

Frequently Asked Questions

What is the difference between LayerNorm and RMSNorm?

LayerNorm subtracts the mean, divides by the standard deviation, then applies a learned scale and shift. RMSNorm skips the mean subtraction and divides by the root mean square only, with a single learned gain. The difference is one reduction and one parameter vector per layer.

Does RMSNorm lose accuracy compared to LayerNorm?

In the original RMSNorm paper it matched LayerNorm on machine translation benchmarks while training faster, and Llama, Gemma, Mistral, and Qwen all use it at scale. At small scale the gap is usually within noise, so speed and simplicity decide it.

Why is pre-norm more stable than post-norm?

Post-norm puts a normalization layer directly on the residual highway, so every gradient has to pass through it at every layer. Pre-norm normalizes only the sublayer input, leaving an unbroken identity path for gradients, which is why deep stacks train without elaborate warmup.

Does pre-norm have a downside?

Yes. Because the residual stream is never re-normalized, activation magnitudes grow with depth at initialization, and large spikes can persist through the residual connections. Real models compensate with a final norm before the output head and, increasingly, with QK-norm or sandwich normalization.

Where does the final norm go in a pre-norm transformer?

After the last block and before the output head. Pre-norm leaves the residual stream un-normalized, so without a final norm the logits are computed from activations whose scale depends on depth.

Sources

  • [1] Zhang and Sennrich, Root Mean Square Layer Normalization — the RMSNorm paper and its WMT comparisons: arxiv.org
  • [2] Xiong et al., On Layer Normalization in the Transformer Architecture — pre-LN versus post-LN gradient behaviour: arxiv.org
  • [3] Pre-LN residual stream growth — analysis of activation magnitude with depth in pre-norm stacks: arxiv.org
  • [4] Peri-LN: Revisiting Normalization Layer in the Transformer Architecture — normalization on both sides of the sublayer: arxiv.org
  • [5] Raschka, Build a Large Language Model (From Scratch) — a full pre-norm GPT implementation, block by block: amazon.com

What are You Looking For?

esc