Sam Austinon November 14, 202613 min read

Scaling Laws Explained: Why Bigger Models Trained on More Data Keep Improving

Scaling Laws Explained: Why Bigger Models Trained on More Data Keep Improving
Contents

Figure 1: Scaling Laws Explained: Why Bigger Models Trained on More Data Keep Improving — a straight line on a log-log plot is the closest thing this field has to a budget forecast

Your team trains a 100M-parameter model, then a 400M one, then a 1.6B one. The loss curve drops in a suspiciously straight line on a log-log plot. Someone asks, "What happens at 100B?" and you realise you can estimate the answer before spending a million dollars.

That is the promise of scaling laws: predictable improvement from predictable investment. This post covers what the two foundational papers actually found, where the field's rules of thumb came from, and what these curves quietly do not promise.

What Is a Scaling Law?

A neural scaling law is an empirical relationship that predicts model performance from scale. In practice it is a power law linking loss to parameters (N), training tokens (D), and compute (C) [1].

The word "empirical" carries the weight. Nobody derived these curves from first principles. Researchers trained many models, plotted the results, and found a pattern that held across several orders of magnitude. It is a trendline that turned out to be astonishingly clean — with a very large GPU bill attached.

The Three Ingredients

Every scaling law talks about three quantities:

  • Parameters (N): the model's size.
  • Tokens (D): how much data it trains on.
  • Compute (C): total training budget in floating-point operations.

The standard conversion is C ≈ 6ND — roughly six FLOPs per parameter per token, covering the forward pass and the backward pass [1]. That one approximation lets you move between model size, data, and cost on the back of an envelope, and it is the reason "compute-optimal" is a meaningful phrase at all.

Kaplan: Loss Falls as a Power Law

The 2020 paper from Kaplan and colleagues at OpenAI started the modern conversation [1]. Cross-entropy loss falls as a power law in compute across more than five orders of magnitude, and the paper fitted separate exponents for parameters, data, and compute.

The headline exponents were roughly 0.076 for parameters and 0.095 for data. Those numbers say something important: loss shrinks slowly. You need large multiples in scale for modest loss reductions. That is exactly why labs keep scaling — small per-step improvements compound into large capability gaps at the frontier.

The paper concluded that with a fixed budget you should grow model size faster than data. That advice shaped the era of ever-larger models trained on relatively little data — GPT-3 being the canonical example, at 175B parameters and 300B tokens [5].

Chinchilla: Train Smaller Models Longer

In 2022, Hoffmann and colleagues at DeepMind re-ran the sweeps and reached a different answer [2]. The corrected recipe: allocate compute roughly equally between model size and data, which works out to about 20 tokens per parameter.

For GPT-3's compute budget, that would have meant a 70B-parameter model trained on 1.4 trillion tokens.

The proof came from a head-to-head. Chinchilla, with 70B parameters and 1.4T tokens, outperformed Gopher's 280B parameters and 300B tokens at the same FLOP budget. The field had been training oversized, undertrained models.

The practical lesson is blunt: a quarter of the parameters and nearly five times the data beat a bigger model at the same cost.

Why Did the Two Papers Disagree?

The disagreement was partly a learning-rate-schedule artifact, not a discovery about intelligence. Hoffmann et al. discuss this directly: when the schedule is tuned per run, the data exponent grows and the optimal allocation moves toward roughly equal split [2].

The lesson generalises beyond scaling laws: experimental details change the conclusion. A baseline that loses is often just an untuned baseline.

Later work has also re-examined the Chinchilla fit and questioned its statistical estimates, so treat the exact constants as a fit on one dataset with one recipe, not as constants of nature. That is the right posture for every number in this post.

The Loss Has a Floor

Scaling laws do not promise infinite gains. The standard parametric form includes an irreducible term:

L(N, D) = E + A / N^α + B / D^β

E is the floor. It represents the entropy inherent in the data — the information no model can predict, like an author's genuinely arbitrary word choice. No amount of compute goes below it, so scaling only shrinks the reducible gap between your current loss and that floor [1][2].

That is why the curve bends. Early on, the power-law terms dominate and the line is straight on log-log axes. Near the floor, the reducible part gets small and further scale buys less and less.

Why "Compute-Optimal" Isn't the Same as "Best"

Here is where practice departs from the papers. Chinchilla answers a narrow question: what is the cheapest way to hit a given training loss? Real products care about serving costs too.

Account for inference and the answer flips. If a model will serve millions of queries, training a smaller model for substantially longer than Chinchilla-optimal is cheaper overall — you overspend once on training so every future query runs on a smaller, faster model.

Llama 3 8B is the clearest public example. Meta trained it on roughly 15 trillion tokens, far past the compute-optimal point for 8B parameters, and the model card describes that as a deliberate choice optimised for inference cost [4].

The arithmetic is simple. A one-time training bill of a few million dollars is trivial next to years of per-query inference across a user base. Compute-optimal is the right target only if you train a model and never serve it.

A Second Axis: Compute at Inference Time

Scaling laws used to describe training only. Newer work adds a second axis: spending compute when the model answers.

  • Repeated sampling: pass@k — the chance that at least one of k samples succeeds — rises roughly log-linearly with the number of samples [5].
  • Thinking longer: reasoning models such as OpenAI's o1 and DeepSeek-R1 show that more inference-time reasoning can trade against pretraining compute.
  • Test-time versus parameters: Snell and colleagues showed that scaling test-time compute optimally can be more effective than scaling model parameters for a fixed budget [3].

Why this matters: it changes the budget math entirely. A model that reasons for longer at answer time is a different purchase than a bigger model, and the two can be compared on the same FLOP ledger.

What Scaling Laws Don't Promise

This section matters most, because scaling laws get misread constantly.

Loss Isn't Capability

The curves predict cross-entropy loss, a smooth aggregate number. They do not directly predict whether a model can do arithmetic, write working code, or follow a tricky instruction. Downstream scores can move abruptly while loss moves smoothly — and whether those jumps are real "emergent abilities" or artifacts of thresholded metrics is still debated.

Measure the task you care about. Loss predicts loss.

Extrapolation Is Risky

Scaling laws interpolate well and extrapolate with risk. Change the architecture, the data mix, the optimizer, or the precision, and the constants move. Fitting on three sizes and predicting two orders of magnitude out is how people ship disappointing runs.

Data Is Finite

High-quality public text is a finite resource, and projections of when frontier-scale training would exhaust it put the date within this decade. Synthetic and multimodal data are the escape route, but whether they scale as cleanly as web text remains open.

Constants Aren't Universal

The exponents depend on the dataset, tokenizer, architecture, and training recipe. Every serious lab refits the ratio before a big run instead of quoting a published one. The 20-to-1 number is a starting point, not a law.

Cost Still Bites

Even a clean curve costs real money, and the whole value of the curve is avoiding a wrong run. Predictions help you spend once instead of three times.

How to Fit Your Own Scaling Law

You do not need a frontier lab to try this. A small team can run a miniature version:

  1. Pick a fixed recipe — same architecture family, data, tokenizer, and optimizer.
  2. Train a grid — several model sizes, each at several token counts.
  3. Tune honestly — use a sensible learning-rate schedule for every run, since sloppy tuning distorts the fit.
  4. Record final validation loss for every run.
  5. Fit the parametric form and inspect the residuals.
  6. Predict a held-out larger run, then train it and see whether the prediction holds.

Here is a minimal fitting sketch for L = E + A/N^α + B/D^β:

import numpy as np
from scipy.optimize import curve_fit

def loss_fn(ND, E, A, alpha, B, beta):
    N, D = ND
    return E + A / N**alpha + B / D**beta

# N, D, L are 1-D numpy arrays from your own training runs
popt, _ = curve_fit(
    loss_fn, (N, D), L,
    p0=[1.7, 400.0, 0.3, 400.0, 0.3],
    maxfev=20000,
)
E, A, alpha, B, beta = popt
print(f"floor E={E:.3f}, alpha={alpha:.3f}, beta={beta:.3f}")

I ran this on synthetic data with known parameters (E=1.7, α=0.34, β=0.28) to see how well it recovers them:

true   [1.7,  400.0, 0.34,  400.0, 0.28]
fit    [1.698, 358.9, 0.332, 476.2, 0.289]
max abs residual: 0.0122

The floor and both exponents come back close. The coefficients A and B do not — they trade off against each other, which is exactly why two people fitting the same data can report different-looking constants and both be right about the exponents.

Fitting is touchy. Different initial guesses can land on different answers, so run several starts, check the residuals, and hold out your largest run as the real test. Otherwise you are curve-fitting noise.

Practical Takeaways

  • Plan with small runs. Forecast before you scale, then verify with a held-out run.
  • Do not treat 20 tokens per parameter as law. Treat it as a starting point for your own fit.
  • Decide who pays inference. Heavy traffic favours a smaller model trained longer.
  • Budget for test-time compute. If you sample many answers or reason step by step, joint training-and-inference budgeting matters.
  • Invest in data quality. At the margin, better data often beats more data.
  • Watch the floor. If your curve flattens, you may be near irreducible loss, or your recipe may have a ceiling.
  • Measure downstream tasks directly. Loss predicts loss, not product quality.

Common Mistakes to Avoid

  • Quoting constants without context. Exponents depend on recipe and data.
  • Extrapolating many orders of magnitude. Validate with a held-out large run.
  • Ignoring tuning. The Kaplan–Chinchilla disagreement is a tuning artifact, not a mystery.
  • Confusing loss with ability. A lower loss does not guarantee a specific skill.
  • Optimising training cost alone. Serving costs often dominate over a model's lifetime.
  • Assuming infinite data. Plan for data limits and quality.
  • Treating any single source as gospel. Go to the papers and refit.

Notice that most of these mistakes involve trusting a clean-looking number too much. A straight line on a log-log plot is evidence, not a guarantee.

Best Book to Read

CoverBookDescriptionGet it
Cover of “AI Engineering” AI Engineeringby Chip Huyen covers model selection, cost and latency tradeoffs, and why inference economics often decide architecture choices that scaling laws alone would not. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Read the chapters on model adaptation and inference cost after this post. They turn the abstract "who pays inference" argument into concrete budgeting, which is the part that actually changes decisions.

Final Thoughts

Scaling laws explain why bigger models trained on more data keep improving: loss falls as a smooth power law in parameters, data, and compute, down to an irreducible floor. Kaplan showed the pattern, Chinchilla corrected the balance, and modern practice adds overtraining for cheaper serving plus test-time compute as a second dial.

The laws guide budgets, but they do not guarantee capabilities, and their constants shift with every change in recipe.

Run a tiny experiment this week: train three model sizes at three data sizes, plot loss on log-log axes, and see whether your points line up. If they do, you have reproduced the idea behind a multi-billion-dollar industry on a laptop. If they do not, you have found out how sensitive these curves are to tuning — which is half the lesson.

Frequently Asked Questions

What is a neural scaling law?

An empirical power-law relationship between a model's loss and its scale, measured by parameters, training tokens, and compute. It is fitted to experimental runs, not derived from theory, so its constants hold only for the recipe and data that produced them.

What is the difference between Kaplan and Chinchilla?

Kaplan et al. concluded that with a fixed compute budget you should grow parameters faster than data. Chinchilla re-ran the sweeps with better-tuned learning-rate schedules and found compute should be split roughly equally, giving about 20 tokens per parameter.

Is 20 tokens per parameter a rule?

No. It is a fit on one data distribution with one training recipe. Modern practice deliberately trains far past it, because a smaller model trained longer is cheaper to serve. Llama 3 8B was trained on roughly 15 trillion tokens, far beyond the Chinchilla point.

Do scaling laws predict capabilities?

They predict cross-entropy loss, which is a smooth aggregate. They do not directly predict whether a model can write working code or follow an instruction, and downstream scores can move abruptly while loss moves smoothly.

Why does loss stop improving?

The standard form includes an irreducible term E, the entropy inherent in the data itself. No amount of compute removes it, so scaling only shrinks the reducible gap between current loss and that floor.

Sources

  • [1] Kaplan et al., Scaling Laws for Neural Language Models — the power-law fits and the C ≈ 6ND estimate: arxiv.org
  • [2] Hoffmann et al., Training Compute-Optimal Large Language Models — Chinchilla, the 20-tokens-per-parameter result, and the learning-rate discussion: arxiv.org
  • [3] Snell et al., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters: arxiv.org
  • [4] Meta AI, The Llama 3 Herd of Models — the 8B model trained on 15T tokens and the inference-cost rationale: arxiv.org
  • [5] Brown et al., Language Models are Few-Shot Learners — GPT-3's 175B parameters and the pass@k scaling result: arxiv.org

What are You Looking For?

esc