Sam Austinon November 14, 202614 min read

Mixture of Experts (MoE) Explained: How Sparse Models Get Big Without Getting Slow

Mixture of Experts (MoE) Explained: How Sparse Models Get Big Without Getting Slow
Contents

Figure 1: Mixture of Experts (MoE) Explained: How Sparse Models Get Big Without Getting Slow — a router decides which slice of a very large model each token is allowed to touch

A model with 671 billion parameters answers your question at roughly the compute cost of a 37 billion parameter one. That sounds like a magic trick, or a marketing lie. It is neither. Mixture of Experts stores a huge amount of knowledge and uses a small slice of it for each token.

The trick is not compression and it is not distillation. It is conditional computation: a small router network looks at each token and picks which few sub-networks get to run. This post separates the three costs people constantly blur together — compute, memory, and communication — and shows a toy MoE layer whose numbers I measured on this machine.

Dense vs. Sparse: The Core Idea

A standard transformer is dense. Every token flows through every parameter in every layer, so doubling the parameters roughly doubles the compute per token.

An MoE model is sparse. The dense feed-forward block is split into several experts, and only a few run per token [1]. Total parameters grow with the number of experts; runtime cost grows only with the number of active ones.

Think of a hospital. It employs hundreds of specialists, but your visit involves two or three. The building is enormous; the labour spent on you is small.

That is the whole reason bigger models do not always cost proportionally more to run. The capacity is real, the compute is rationed.

How an MoE Layer Works

Most MoE transformers keep attention dense and replace the feed-forward network (FFN) in some or all blocks with an MoE layer. That layer has two parts:

  • Experts: several independent FFNs, each with its own weights.
  • Router (gate): a small network that scores each expert for each token.

The recipe, in order:

  1. Score — the router produces one score per expert for the token.
  2. Select — the top-k experts win.
  3. Compute — the token runs through only those experts.
  4. Combine — their outputs merge, weighted by the router scores.

The settings vary by model. Mixtral picks 2 of 8 experts per token [3]. Switch Transformer picks just 1, which is the simplification that made trillion-parameter sparsity practical [2]. DeepSeek-V3 uses sigmoid affinities rather than softmax for its routed experts, and adds a shared expert that every token passes through [4].

Total vs. Active Parameters

This distinction explains almost everything about MoE. Total parameters measure how much the model stores. Active parameters measure how much it uses per token.

The published figures, from the labs' own reports and model cards:

Model Total Active Active share Experts
Mixtral 8x7B 46.7B 12.9B 27.6% 8, top-2
Mixtral 8x22B 141B 39B 27.7% 8, top-2
Llama 4 Scout 109B 17B 15.6% 16
Llama 4 Maverick 400B 17B 4.3% 128
DeepSeek-V3 671B 37B 5.5% 256 routed + 1 shared
Qwen3-30B-A3B 30.5B 3.3B 10.8% 128, top-8

Notice the ordering: the largest MoEs activate the smallest fraction of their weights. DeepSeek-V3 runs about one nineteenth of its parameters per token. A bigger model can afford to wake a smaller share of itself, because the absolute number of active parameters is what the compute bill tracks.

A warning about sloppy numbers. Sources disagree on Mixtral: some say 45B total, others 47B, and some list 11.2B active with top-1 routing. Mixtral uses top-2 routing, and Mistral's own figures are 46.7B total and 12.9B active [3]. Always go to the model card, because secondary sources copy each other's errors.

Why It Gets Big Without Getting Slow

Compute per token tracks active parameters, not total ones. For a decoder-only transformer, approximate FLOPs scale linearly with the activated parameters, so Mixtral 8x22B at 141B total and 39B active costs roughly like a 39B dense model per token while holding far more capacity [3].

Training benefits too. Switch Transformer reported up to 7x faster pretraining at the same compute budget compared with a dense baseline of matched cost [2]. MoE bends the scaling-law tradeoff: you buy more parameters per FLOP. Those gains come with conditions — routing must stay balanced and the experts must fit — which is where the rest of this post goes.

A Short History

  • Early 1990s: Jacobs, Jordan, Nowlan, and Hinton publish "Adaptive Mixtures of Local Experts", the original mixture-of-experts idea with a gating network.
  • 2017: Shazeer and colleagues bring sparsely gated MoE to language models, routing each token to its top-k experts and adding a load-balancing loss [1].
  • 2020–2021: GShard and Switch Transformer scale the idea to billions of parameters, and Switch simplifies routing to one expert per token [2].
  • 2024: Mixtral ships strong open-weight MoE checkpoints [3], and DeepSeek-V3 scales the design to 671B total parameters [4].
  • 2025 onward: Llama 4 and Qwen3 adopt MoE across their model families.

Vendor claims about which closed frontier models are MoE should be treated as reports, not facts. Several labs have never published their architectures.

The Hardest Problem: Load Balancing

Left alone, routers play favourites. The hardest part of MoE training is stopping the router from sending nearly every token to the same two experts. That is expert collapse: popular experts absorb all the tokens while the rest stop receiving gradient and atrophy.

Fix 1: Auxiliary Load-Balancing Loss

The standard fix adds a penalty for uneven routing. The Switch-style version multiplies each expert's token fraction by its mean routing probability and sums across experts, scaled by the expert count so a perfectly uniform router scores 1.0 [2]:

aux = N * Σ_i ( fraction_of_tokens_to_expert_i × mean_router_prob_for_expert_i )

Typical coefficients sit between 0.01 and 0.1. Too strong a penalty distorts the main objective; too weak lets collapse happen. This is a dial, not a fixed constant.

Fix 2: Auxiliary-Loss-Free Balancing

DeepSeek-V3 takes a different route. Instead of a loss term, it keeps a per-expert bias that rises for underloaded experts and falls for overloaded ones, then uses that bias when selecting the top-k [4]. Load balances without letting a penalty term interfere with the language-modelling objective.

Fix 3: Shared Experts and Fine-Grained Experts

DeepSeek-V3 also uses one always-on shared expert plus 256 routed experts, selecting 8 per token [4]. The shared expert captures patterns every token needs, so the router does not have to duplicate basic skills across many experts. Many small experts plus a few shared ones is the design pattern most 2025-era open models converged on.

Other Stabilisers

  • Capacity limits cap how many tokens each expert accepts; overflow tokens are dropped or rerouted depending on the implementation.
  • Router z-loss penalises extreme router logits, which otherwise destabilise training [5].
  • Dropless execution lets experts take whatever load arrives, trading memory for stability.

Check each paper for the exact formulation, because the details differ more than the summaries suggest.

The Catches

This is where "big without slow" gets honest.

Memory

All experts must live in memory even though only a few run. Mixtral's 46.7B parameters need 93.4 GB at 16-bit precision (46.7 billion × 2 bytes), or 87 GiB, before activations and KV cache. You pay compute for 13B and memory for 47B. Quantisation helps, but it does not remove the gap.

Latency Is Not Proportional

Fewer active parameters does not automatically give proportionally lower latency. The reasons:

  • Grouping tokens by expert adds gather and scatter overhead around the actual matrix multiplies.
  • Imbalanced loads leave some GPUs waiting while others finish, so the slowest expert sets the step time.
  • Batching changes the picture. With a larger batch, tokens cover more distinct experts, so more weights get touched per step — the saving shrinks as throughput grows.

The compute saving is real; the wall-clock saving is smaller and depends on your batch size and kernel quality.

Communication

In large deployments, experts live on different GPUs, so tokens travel between them. That is expert parallelism with all-to-all routing, and GPU-to-GPU communication is a first-order bottleneck, not a detail. Slow interconnects can erase the compute savings entirely, which is why MoE training clusters are built around fast fabrics.

Fine-Tuning and Stability

Routers can behave unstably early in training, and fine-tuning an MoE behaves differently from fine-tuning a dense model. Monitor the expert utilisation histogram from step one, and raise the auxiliary coefficient if one expert hogs the traffic.

Experts Are Not Topic Specialists

A common myth says each expert becomes "the maths expert" or "the coding expert". In practice experts pick up subtler patterns, which makes them hard to interpret and hard to prune. You cannot delete a skill by deleting an expert.

A Toy MoE Layer in PyTorch

Here is a minimal implementation of the moving parts. It skips capacity limits and efficient batching, so treat it as teaching code:

import torch, torch.nn as nn, torch.nn.functional as F

class MoELayer(nn.Module):
    def __init__(self, d_model=256, d_ff=1024, n_experts=8, k=2):
        super().__init__()
        self.k, self.n = k, n_experts
        self.router = nn.Linear(d_model, n_experts, bias=False)
        self.experts = nn.ModuleList([
            nn.Sequential(nn.Linear(d_model, d_ff), nn.GELU(),
                          nn.Linear(d_ff, d_model))
            for _ in range(n_experts)
        ])

    def forward(self, x):                        # x: (tokens, d_model)
        probs = F.softmax(self.router(x), dim=-1)         # (T, E)
        topv, topi = probs.topk(self.k, dim=-1)           # (T, k)
        topv = topv / topv.sum(-1, keepdim=True)          # renormalise weights
        out = torch.zeros_like(x)
        for e in range(self.n):
            tok, slot = (topi == e).nonzero(as_tuple=True)
            if tok.numel():
                out[tok] += topv[tok, slot, None] * self.experts[e](x[tok])
        # Switch-style balancing loss (using each token's top-1 choice)
        frac = F.one_hot(topi[:, 0], self.n).float().mean(0)
        aux = self.n * (frac * probs.mean(0)).sum()
        return out, aux

layer = MoELayer()
x = torch.randn(512, 256)
y, aux = layer(x)

Measured output on this machine:

out shape: (512, 256)
aux loss at init: 1.0033        (uniform routing scores exactly 1.0)
total params: 4,206,592
per-expert params: 525,568
router params: 2,048
active per token (k=2): 1,053,184 = 25.0% of total

The accounting is the lesson. With 8 experts and top-2 routing, a quarter of the weights run per token — the same ratio Mixtral shows at 27.6%. Sparsity is arithmetic, not magic.

The Collapse Experiment Did Not Reproduce

I tried the obvious experiment: train this layer on a small synthetic regression task with the auxiliary coefficient set to 0, 0.01, and 0.1, then count how many tokens each expert receives. Over 400 steps with a batch of 256, the shares came out:

aux coef 0.0   expert share % [12.0, 15.8, 13.8, 13.5, 11.2, 11.3, 14.6, 7.8]  aux=1.0055  mse=0.0369
aux coef 0.01  expert share % [12.2, 14.3, 13.7, 12.8, 11.8, 11.8, 13.0, 10.5]  aux=1.0010  mse=0.0369
aux coef 0.1   expert share % [12.1, 13.5, 13.3, 12.9, 11.9, 11.7, 12.7, 11.9]  aux=1.0007  mse=0.0370

Uniform routing would be 12.5% each, and that is roughly what I got — with no auxiliary loss at all. Expert collapse did not appear on a toy task.

That is worth knowing rather than hiding. Collapse is a scale and pressure phenomenon: it shows up when routing decisions compound over many layers, when the task is hard enough that some experts genuinely fit better, and when capacity limits force competition. A single layer, an easy target, and softmax over 8 experts give the router no reason to concentrate. The toy code teaches the mechanics; it does not reproduce the pathology.

The real diagnostic is the utilisation histogram during training, logged from step one, plus a held-out evaluation that would catch a router quietly ignoring most of its capacity.

MoE vs. Dense: When to Choose Which

Choose MoE when:

  • You serve at scale — per-token compute savings compound across millions of requests.
  • You have memory and fast interconnects — all experts must fit, and all-to-all traffic must stay cheap.
  • You want capacity per training FLOP — the sparse design buys parameters cheaply.
  • You can absorb the engineering — routing, balancing, and parallelism are real work.

Choose dense when:

  • You run on one GPU or edge hardware — memory limits kill the advantage.
  • You fine-tune often on small data — dense models behave more predictably.
  • You want simple serving — dense inference stacks are mature.
  • Your team is small — operational complexity can outweigh the savings.

Many teams never train an MoE. They download an open-weight one and serve it, which makes the practical question: does my serving stack handle this model's memory footprint and routing?

How to Evaluate an MoE Model Fairly

Do not compare on active parameters alone. Run this protocol:

  1. Compare at equal quality, then measure cost — not at equal parameter count.
  2. Measure memory — total weights plus KV cache at your real context length.
  3. Measure throughput and latency at your real batch sizes.
  4. Test on your own tasks, not just leaderboards.
  5. Check quantised versions, since memory often decides feasibility.
  6. Watch tail latency, because routing imbalance creates stragglers that averages hide.

Common Pitfalls

  • Comparing active parameters to dense total parameters. Compare at matched quality and cost.
  • Ignoring memory. The model still needs every expert resident.
  • Assuming linear latency gains. Communication, batching, and imbalance change the maths.
  • Skipping load-balancing diagnostics. Log expert utilisation from the first step.
  • Over-tuning the balancing loss. Too much hurts quality; too little invites collapse.
  • Believing "expert = topic". Experts rarely map to human categories.
  • Trusting blog numbers. Parameter counts drift between secondary sources; use model cards.
  • Forgetting fine-tuning costs. Adapting an MoE is not the same job as adapting a dense model.

Notice that most of these mistakes come from confusing compute with memory. The FLOPs are cheap; the bytes are not.

Best Book to Read

CoverBookDescriptionGet it
Cover of “Build a Large Language Model (From Scratch)” Build a Large Language Model (From Scratch)by Sebastian Raschka builds a transformer block by block in PyTorch, which is exactly the level of understanding you need before replacing its FFN with a routed one. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Read the attention and FFN chapters first, then come back to this post. The MoE layer above is a drop-in replacement for the feed-forward block you will have just written, and the parameter accounting becomes obvious once you have counted the weights yourself.

Wrapping Up

MoE works because compute follows active parameters while capacity follows total parameters. That gap lets labs build enormous models that run at modest per-token cost — as long as they handle routing balance, memory, and communication. Dense models stay simpler and friendlier on small hardware; MoE shines when you serve at scale.

The numbers to remember: DeepSeek-V3 stores 671B and computes 37B, Mixtral stores 46.7B and computes 12.9B, and the memory bill is charged on the stored figure at 2 bytes per parameter.

Build the toy layer above this week and log the utilisation histogram. If one expert hogs the traffic, you have reproduced expert collapse. If the load spreads evenly — as mine did on a toy task — you have learned something just as useful: that the pathology needs scale to appear.

Frequently Asked Questions

What is the difference between total and active parameters in an MoE?

Total parameters measure storage: every expert must be loaded in memory. Active parameters measure compute: only the experts a token is routed to are multiplied against it. DeepSeek-V3 has 671B total and 37B active, so it stores a huge amount but computes roughly like a 37B dense model per token.

Why are MoE models not proportionally faster?

Because compute is only one of three costs. All experts still occupy memory, routing adds gather and scatter overhead, load imbalance leaves GPUs idle, and in multi-GPU deployments tokens must travel between devices over all-to-all links. A fast interconnect and balanced routing are what turn the theoretical saving into real latency.

What is expert collapse and how do you prevent it?

Collapse is when the router sends most tokens to a handful of experts while the rest stop receiving gradient and atrophy. The standard fix is an auxiliary load-balancing loss that penalises uneven routing. DeepSeek-V3 instead uses per-expert bias terms that rise for underloaded experts and fall for overloaded ones, avoiding a second loss term.

Do MoE experts specialise by topic?

Rarely in the human sense. Experts tend to pick up subtle syntactic and distributional patterns rather than clean categories like maths or code. That makes them hard to interpret and hard to prune, and it means you cannot delete an expert by naming the skill you want removed.

Should I use an MoE model or a dense one?

If you serve high traffic on hardware with enough memory and fast interconnects, an MoE gives more capacity per FLOP. If you run on one GPU, fine-tune often on small data, or want a simple serving stack, a dense model is the safer engineering choice.

Sources

  • [1] Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — top-k gating and the load-balancing loss: arxiv.org
  • [2] Fedus, Zoph, and Shazeer, Switch Transformers — one-expert routing, the 7x pretraining speedup, and the auxiliary loss form: arxiv.org
  • [3] Jiang et al., Mixtral of Experts — 46.7B/12.9B and 141B/39B with top-2 routing: arxiv.org
  • [4] DeepSeek-AI, DeepSeek-V3 Technical Report — 671B total, 37B active, 256 routed plus 1 shared expert, sigmoid affinities, and auxiliary-loss-free balancing: arxiv.org
  • [5] Zoph et al., ST-MoE: Designing Stable and Transferable Sparse Expert Models — router z-loss and the stability toolkit: arxiv.org

What are You Looking For?

esc