Sam Austinon November 13, 202612 min read

SGD vs Adam vs AdamW: Which Optimizer Should You Use and Why?

SGD vs Adam vs AdamW: Which Optimizer Should You Use and Why?
Contents

Figure 1: SGD vs Adam vs AdamW: Which Optimizer Should You Use and Why? — three ways to take the same step, and one line of code that changes which one you actually get

You open a training script and see optim.Adam, optim.AdamW, or optim.SGD. Most tutorials say "use Adam" and move on, and that advice has a catch.

The catch is not that Adam is bad. It is that optim.Adam(..., weight_decay=0.01) does not do what almost everyone thinks it does, and the fix has a different name.

The Short Answer

If you read nothing else:

  • AdamW — start here for transformers, LLM fine-tuning, and most new projects.
  • SGD with momentum — the right call for classic vision models when you can afford a proper learning rate sweep.
  • Adam — only when a repository or paper requires it, because with decay enabled it is the broken version of AdamW.

The rest of this article is why, and the why is more interesting than the ranking.

What Actually Differs: The Three Updates

All three take a step of size α in a direction derived from the gradient. The differences are in how the direction is built and where the decay enters.

SGD with momentum keeps one buffer and smooths the gradient:

v ← βv + g, θ ← θ − αv

Adam keeps two buffers, one for the mean and one for the squared gradient, and divides by the second:

m ← β₁m + (1−β₁)g v ← β₂v + (1−β₂)g² θ ← θ − α · m̂ / (√v̂ + ε)

where m̂ and v̂ are bias-corrected by 1−β₁ᵏ and 1−β₂ᵏ.

AdamW is the same two buffers, plus one change:

θ ← θ − α · m̂ / (√v̂ + ε) then θ ← θ · (1 − αλ)

That final multiplication is the whole idea. The decay is applied to the weights directly, not added to the gradient.

The One Fact That Explains Everything

Adam with weight_decay=0 and AdamW with weight_decay=0 are the same algorithm. Identical. Bit for bit, up to floating point.

So "Adam vs AdamW" is not a choice between two optimizers. It is a choice about where the weight decay goes — and only matters when you turn decay on.

That reframes the common advice. "Skip Adam, use AdamW" really means: if you are using weight decay, use the version that applies it correctly. If your weight_decay is zero, the distinction disappears.

And here is the trap that catches people. In PyTorch, AdamW defaults to weight_decay=1e-2 and Adam defaults to weight_decay=0 [5]. So:

optim.AdamW(model.parameters(), lr=3e-4)                      # decay 0.01
optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.01)    # decay 0.01, but L2

Those two lines look equivalent and are not. The first shrinks every weight by the same proportion each step. The second pushes an L2 term through the adaptive scaling, and the effective shrinkage varies from parameter to parameter [1].

Why L2 and Weight Decay Are Not the Same Thing

Start with plain SGD, where they genuinely are the same. Add (λ/2)‖θ‖² to the loss and the gradient gains λθ:

θ ← θ − α(g + λθ) = (1 − αλ)θ − αg

That is exactly weight decay: every weight is multiplied by 1 − αλ before the gradient step. Same update, two descriptions. This equivalence is why the two terms get used interchangeably in older code.

Now do it with Adam. The λθ term enters the gradient, so it joins the running averages:

m ← β₁m + (1−β₁)(g + λθ) v ← β₂v + (1−β₂)(g + λθ)²

and then the whole thing is divided by √v̂. The decay term is now scaled by 1/√v̂, which is per-parameter. A weight with a large gradient history gets a large √v̂, so its decay is divided down. A weight with a small gradient history gets a small √v̂, so its decay is amplified.

The regularization you set is not the regularization you get. It is distorted by the same mechanism that makes Adam useful in the first place.

AdamW removes the term from the gradient entirely. The decay is applied after the adaptive step, so every weight shrinks by the same αλ regardless of its gradient history. That is what "decoupled" means, and it is the only difference [1].

What the AdamW Paper Actually Shows

Loshchilov and Hutter did not just propose a cleaner update. They explained a puzzle: for years, Adam generalized worse than SGD with momentum on image classification, and nobody knew why.

Their diagnosis was that common implementations only had L2 regularization, not true weight decay — so Adam was being compared against SGD-with-real-decay while running a distorted penalty. The comparison was rigged without anyone intending it.

The numbers they report:

  • 15% relative improvement in test error over Adam with L2 on CIFAR-10 [1].
  • The optimal weight decay becomes independent of the learning rate, for both SGD and Adam. You can tune two knobs separately instead of one tangled pair.
  • Adam becomes competitive with SGD with momentum on image classification, which it had previously lost to.

One honest caveat, which the authors state themselves: their evidence is image classification, and they flag that a wider range of tasks still needed checking. Since then, the field answered that question for them — AdamW is the default in essentially every modern LLM training stack.

The Memory Bill

SGD with momentum stores one extra tensor per parameter. Adam and AdamW store two. That sounds trivial until the model is large.

For a 7-billion-parameter model in fp32:

Item Size
Weights 28 GB
Gradients 28 GB
Adam/AdamW states (m, v) 56 GB
Total before activations 112 GB
SGD + momentum states 28 GB
Total before activations 84 GB

The optimizer states are larger than the weights. That is why "just use AdamW" stops being free at scale, and why 8-bit optimizer states, sharded optimizers, and CPU offloading exist as a category.

On a small model the difference is noise. On a 70B model it is the difference between fitting and not fitting.

When SGD Still Wins

Convolutional networks on ImageNet are still SGD territory. The standard ResNet recipes — lr=0.1, momentum 0.9, weight decay 5e-4, cosine schedule, long training — are tuned to the point where reproducing them is easier than beating them.

The folklore that "SGD generalizes better" needs an asterisk, though. A large part of that gap was the Adam-with-L2 artifact described above. With matched tuning and correct decay, AdamW is competitive, and the remaining difference is smaller than the old blog posts suggest [1].

SGD also wins on memory, and on any problem where the gradient scales across layers are similar. When every layer wants roughly the same step size, adaptivity buys you little and costs you two buffers.

When AdamW Wins

Transformers, and therefore almost everything current. Attention and embedding gradients live on very different scales, and a single global learning rate handles that badly. Adaptive per-parameter scaling is not a luxury there — it is what makes the model trainable at all.

Fine-tuning. A pretrained model needs small updates to most weights and larger ones to a new head. AdamW's per-parameter scaling does that without a hand-built layer-wise schedule.

Prototyping. AdamW with lr=3e-4 usually produces a falling loss curve within minutes. SGD often needs a sweep before it produces anything useful. When the goal is to find out whether an idea works, that difference dominates.

Starter Settings I Would Use

These are rules of thumb, not guarantees. Check the repository's config first — it usually encodes someone's hard-won tuning.

import torch

# SGD with momentum: classic vision models (ResNet-style recipes)
opt = torch.optim.SGD(
    model.parameters(),
    lr=0.1,
    momentum=0.9,
    weight_decay=5e-4,
)

# AdamW: transformers, fine-tuning, most new projects
opt = torch.optim.AdamW(
    model.parameters(),
    lr=3e-4,
    betas=(0.9, 0.95),
    weight_decay=0.1,
)

A few practical notes that matter more than the numbers:

  • Use a schedule. Warmup plus cosine decay is the default for transformers, and the AdamW paper shows Adam benefits substantially from a scheduled multiplier [1].
  • Lower the rate for fine-tuning. 1e-5 to 5e-5 is normal for a pretrained model; 3e-4 is for training from scratch.
  • Skip weight decay on biases and normalization layers. Many strong recipes do, and it costs nothing to implement with parameter groups.
  • Clip the gradient norm at 1.0 for transformers. It is cheap insurance against a single bad batch.
  • Change one thing at a time. Otherwise you cannot attribute the improvement.

Beyond the Big Three: Muon and SOAP

The optimizer landscape is moving again, so it is worth knowing what is out there — and why it probably is not for your fine-tuning job yet.

Muon

Muon orthogonalises the momentum update for matrix-shaped hidden weights, which spreads the update evenly across directions instead of letting a few dominate. It is not a drop-in replacement: embeddings, classifier heads, and one-dimensional parameters still use AdamW [2].

The scaling result is real. Kimi's scaling-law experiments report that Muon reaches about 2× the computational efficiency of AdamW under compute-optimal training [3]. To get there they had to add two things: weight decay, which the original implementation lacked, and a per-parameter update scale. Even the challenger borrows from AdamW.

Kimi K2 then trained a 1-trillion-parameter mixture-of-experts model with a variant called MuonClip and reported zero loss spikes across 15.5 trillion tokens. That is the strongest evidence any new optimizer has right now — but it is evidence from a frontier lab with a frontier compute budget.

SOAP

SOAP runs Adam in the eigenbasis of a Shampoo preconditioner. In language model pretraining at 360M and 660M parameters, in the large-batch regime, it cuts iterations by over 40% and wall-clock time by over 35% compared to AdamW [4].

Both of these are pretraining results. They assume you can afford the extra state, the extra hyperparameters, and the engineering. For a fine-tuning job on one GPU, AdamW is still the correct answer.

Common Pitfalls

  • Using Adam with a large weight_decay. You are running a distorted penalty. Switch to AdamW.
  • Assuming AdamW(wd=0.01) and Adam(wd=0.01) are the same. They are not. Only wd=0 makes them identical.
  • Copying SGD's learning rate into AdamW. 0.1 and 3e-4 are three orders of magnitude apart for a reason.
  • Forgetting warmup on transformers. The first few hundred steps are where runs die.
  • Comparing optimizers without tuning each one. An untuned baseline is not a baseline, it is a straw man.
  • Changing the optimizer to fix a data problem. Duplicate rows and mislabeled classes do not care which update rule you use.
  • Adding weight decay to biases and LayerNorm. It is usually harmful and always pointless.
  • Trusting the default weight_decay. PyTorch's AdamW default is 0.01; the LLM recipe is often 0.1. They are not interchangeable.

The single highest-value check: open your training script and look at whether it says Adam or AdamW. If it says Adam and weight_decay is non-zero, you have found a silent bug.

Best Book to Read

CoverBookDescriptionGet it
Cover of “Dive into Deep Learning” Dive into Deep Learningby Aston Zhang, Zachary Lipton, Mu Li, and Alexander Smola the optimization chapter derives SGD, momentum, AdaGrad, RMSProp, and Adam from the update rules up, with runnable code for each. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Read the optimization algorithms chapter after you have trained one model end to end. The derivations land much harder once you have watched a loss curve misbehave. The full book is free online at d2l.ai, which is how most people actually read it.

Wrapping Up

AdamW is Adam with the weight decay moved out of the gradient. That is the entire difference, and it is the difference between a penalty you set and a penalty you get.

The evidence is not subtle: a 15% relative test error improvement over Adam with L2, and weight decay that no longer fights the learning rate for control of the same knob [1].

SGD with momentum still earns its place on classic vision tasks, especially when a well-tuned recipe already exists. Adam is now mostly a legacy spelling — and if you find it with a non-zero weight_decay, you have found something to fix.

Next time you open a training script, check the optimizer line before you check anything else. The math behind the update is in gradient descent from derivative to update rule, and the reason weight decay is a prior in disguise is in MLE vs MAP. If you are about to fine-tune something, the tooling comparison in the best fine-tuning repositories is the next stop.

Now go check which optimizer you are actually running.

Frequently Asked Questions

Is AdamW just Adam with weight decay?

No. AdamW applies weight decay directly to the weights, outside the adaptive step, while Adam adds the L2 penalty to the gradient where it gets divided by the square root of the second moment. With weight_decay=0 the two are mathematically identical, so the difference only appears when you actually use decay.

Why does Adam with L2 regularization generalize worse than AdamW?

The L2 gradient term is scaled by one over the square root of the running second moment, so weights with large gradients receive less shrinkage than the coefficient you set implies. The regularization you asked for is not the regularization you get. AdamW removes the term from the gradient and shrinks the weights directly, so the decay is uniform.

Should I use SGD or AdamW for a ResNet?

Either can work, but SGD with momentum and a long schedule is still the standard ImageNet recipe and is well tuned in most repositories. AdamW is competitive when tuned, and much faster to a decent result. If you are not going to run a learning rate sweep, AdamW is the safer default.

What learning rate should I use with AdamW?

Start at 3e-4 for training from scratch and 1e-5 to 5e-5 for fine-tuning a pretrained model. For LLM pretraining the common recipe is 3e-4 or lower with betas of 0.9 and 0.95, weight decay 0.1, gradient clipping at 1.0, and a warmup followed by cosine decay.

What is Muon and should I use it?

Muon orthogonalises the momentum update for matrix-shaped hidden weights, and uses AdamW for embeddings, heads, and one-dimensional parameters. It reports roughly twice the compute efficiency of AdamW in compute-optimal scaling experiments, but it is a pretraining tool. For fine-tuning, AdamW remains the safe choice.

Sources

  • [1] Loshchilov and Hutter, Decoupled Weight Decay Regularization (ICLR 2019) — the L2-versus-decay inequivalence, the 15% result, and the hyperparameter decoupling: arxiv.org/abs/1711.05101
  • [2] Jordan, Jin, Boza, You, Cesista, Newhouse, and Bernstein, Muon: An Optimizer for Hidden Layers in Neural Networks — the reference implementation and the AdamW-for-everything-else rule: github.com/KellerJordan/Muon
  • [3] Kimi, Muon is Scalable for LLM Training (2025) — the weight-decay and update-scale fixes, and the ~2× compute-efficiency result: arxiv.org/abs/2502.16982
  • [4] Vyas, Morwani, Zhao, Brandfonbrener, Shapira, and Kakade, SOAP: Improving and Stabilizing Shampoo using Adam — over 40% fewer iterations and 35% less wall-clock than AdamW at 360M–660M: arxiv.org/abs/2409.11321
  • [5] PyTorch, torch.optim.AdamW — the reference implementation, defaults, and the decoupled decay term: pytorch.org/docs/stable/generated/torch.optim.AdamW.html

What are You Looking For?

esc