Contents
Figure 1: Gradient Descent Math Step by Step: From Derivative to Update Rule — the loss curve you stare at is the shadow of a landscape, and the update rule is how you walk down it
You have seen the formula a thousand times: θ ← θ − α∇L(θ). You have coded it, tuned the learning rate, and watched the loss curve do something reasonable.
But why does it work? Not "it goes downhill" — why does subtracting the gradient decrease the loss, and what is α actually doing?
I once watched a senior engineer stare at a diverging loss curve and say, "the math says this should work." The math does work. But only if you know which assumption failed, and the answer is almost always a number you never computed.
The Core Question: How Do We Decrease a Function?
You are at a point θ₀ on a loss surface and you want to move to θ₁ where L(θ₁) < L(θ₀).
That is two questions, and the whole subject is the answer to them: which direction, and how far.
Step 1: The Directional Derivative
The directional derivative is the rate of change of L when you move along a unit vector v. It is defined as the limit of the difference quotient, and for a differentiable function it equals the gradient dotted with the direction:
D_v L(θ) = ∇L(θ)ᵀ v = ‖∇L(θ)‖ ‖v‖ cos φ
where φ is the angle between the gradient and v. With ‖v‖ = 1:
D_v L(θ) = ‖∇L(θ)‖ cos φ
To decrease L you need this to be negative, which means cos φ < 0, which means the angle is between 90° and 270°. Every direction in that half-plane is a descent direction — the gradient only tells you which half of the space to move into.
The most negative value occurs at φ = 180°, where cos φ = −1. There the directional derivative equals −‖∇L‖, which is the smallest value it can take for a unit vector. So:
v* = −∇L(θ) / ‖∇L(θ)‖
That is the proof that the negative gradient is the direction of steepest descent — not merely a descent direction, but the one with the fastest instantaneous rate [1].
One qualification that matters later: "steepest" is defined relative to a norm. This derivation uses the Euclidean norm. Change the norm and the steepest direction changes — under the L∞ norm it is the sign of the gradient, and under a preconditioner it is −P⁻¹∇L. That is why Newton's method, which uses the Hessian as a preconditioner, takes different steps from gradient descent despite both being "descent methods".
Step 2: What α Actually Does
Direction is settled. Now the distance.
The normalised update would be θ − α∇L/‖∇L‖, which moves exactly α units. The standard update drops the normalisation:
θ₁ = θ₀ − α∇L(θ₀)
The gradient's magnitude is now part of the step length. A large gradient gives a large step at the same α; a tiny gradient gives a tiny step.
And this is where a common description goes wrong. α does not control "direction alignment and step length". It controls step length only. The direction is fixed by the gradient, full stop. If your update points somewhere unhelpful, changing α will not fix it — you need a different direction, which is exactly what momentum, preconditioning, and adaptive methods provide.
There is a practical consequence hiding here. Rescaling the loss rescales every gradient by the same factor, so it rescales the effective step. Multiply your loss by 100 and your α = 0.01 now behaves like α = 1. That is one of the most common causes of "the same code diverges on my dataset".
Step 3: The Taylor Expansion — Why Subtracting Works
Expand L around θ in the direction −α∇L. The second-order Taylor expansion is:
L(θ − α∇L) = L(θ) − α‖∇L‖² + (α²/2) ∇Lᵀ H ∇L + O(α³)
where H is the Hessian. The first-order term is strictly negative for any α > 0, because ‖∇L‖² ≥ 0. The second-order term can go either way, depending on the curvature along the gradient direction.
So the first-order term decreases the loss and the second-order term may fight it. For a small enough α the first term wins, and the update descends. That is the hand-wave. Here is the exact version.
The descent lemma gives you a number
Assume the gradient is L-Lipschitz: ‖∇L(x) − ∇L(y)‖ ≤ L‖x − y‖ for all x, y. That single assumption bounds the second-order term, and the bound is:
L(θ − α∇L) ≤ L(θ) − α(1 − αL/2) ‖∇L‖²
Read the bracket. The loss decreases whenever α(1 − αL/2) > 0, which holds for:
0 < α < 2/L
and the decrease is monotone and clean for α ≤ 1/L. At α = 1/L the step reduces the loss by exactly ‖∇L‖²/(2L).
So "sufficiently small" has an answer: 2/L. For a quadratic with Hessian A, L is the largest eigenvalue λ_max, which is why the stability limit you meet in practice is α < 2/λ_max. This is the number the engineer in the opening story never computed.
Step 4: Choosing the Step Size
Exact line search minimises φ(α) = L(θ + αd) along the descent direction d = −∇L. For a quadratic it has a closed form; for everything else it is an inner optimisation problem, and it is usually not worth it.
Backtracking line search is the practical version. Start with α = 1 and shrink until the step satisfies the Armijo condition:
L(θ + αd) ≤ L(θ) + c₁ α ∇Lᵀd
with c₁ typically 1e-4. This is sufficient decrease: it demands a fraction of the decrease the linear model predicts, which rules out steps that barely move.
The Wolfe conditions add a curvature condition:
|∇L(θ + αd)ᵀ d| ≤ c₂ |∇Lᵀ d|
with c₂ typically 0.9. The curvature condition stops the search from accepting a tiny step just because it satisfies Armijo — it insists the slope has actually flattened.
In deep learning nobody runs a line search, because each evaluation costs a full forward pass over a large dataset. Fixed schedules, cosine decay, and adaptive methods replace it. The theory still tells you what those methods are approximating.
Step 5: Convergence — When and How Fast
Gradient descent stops when ∇L(θ) ≈ 0. That is a stationary point, and it can be a local minimum, a saddle, or a local maximum.
For convex functions, every stationary point is a global minimum — but convexity alone does not guarantee that gradient descent finds one. Two things must also hold: a minimiser has to exist, and the step size has to be small enough. A convex function can have no stationary point at all (L(θ) = e^θ), and a convex quadratic diverges happily if α > 2/λ_max. Convexity gives you the landscape; the step size decides whether you can walk on it.
The rates
For convex L with L-Lipschitz gradient, fixed step α ≤ 1/L:
L(θₖ) − L(θ*) ≤ ‖θ₀ − θ*‖² / (2αk)
That is O(1/k) — sublinear. To halve the gap you need twice as many iterations.
For μ-strongly convex L:
‖θₖ − θ*‖² ≤ (1 − αμ)ᵏ ‖θ₀ − θ*‖²
That is linear convergence — the error shrinks by a constant factor each step, so the log-error is a straight line. With α = 1/L the factor is 1 − μ/L.
The ratio κ = L/μ is the condition number, and it sets the speed. A well-conditioned problem (κ ≈ 1) converges in a handful of steps. A badly conditioned one (κ = 10⁶) needs on the order of κ iterations, which is why ill-conditioned problems are the ones that eat your GPU budget.
Step 6: The Geometry — What the Landscape Looks Like
The quadratic bowl, solved exactly
Take L(θ) = ½θᵀAθ − bᵀθ + c, so ∇L = Aθ − b and H = A. Gradient descent becomes:
θₖ₊₁ = θₖ − α(Aθₖ − b) = (I − αA)θₖ + αb
Convergence depends entirely on the eigenvalues of (I − αA). Each one is 1 − αλᵢ, and the iteration converges only if |1 − αλᵢ| < 1 for every eigenvalue — which is exactly 0 < α < 2/λ_max.
The optimal fixed step is α* = 2/(λ_min + λ_max), and the resulting per-step contraction factor is:
(κ − 1) / (κ + 1)
Note what this formula says: with κ = 100, the factor is 0.98, so you need roughly 230 iterations to reduce the error by 100×. That is the price of conditioning, and no amount of learning-rate tuning removes it.
Ravines
A ravine is a quadratic with one huge eigenvalue and one tiny one. The steep wall and the flat floor want completely different step sizes. Pick α for the wall and you crawl along the floor; pick it for the floor and you oscillate up and down the wall. The zigzag you see in a loss curve is this, drawn.
Momentum helps. Adaptive methods help. Preconditioning helps most — dividing by the curvature is the only thing that fixes the ratio itself rather than working around it.
Step 7: Momentum and Nesterov Acceleration
Plain gradient descent has no memory. It recomputes a direction from scratch at every step, so it re-derives the same sideways component over and over in a ravine.
Momentum accumulates a velocity:
vₖ₊₁ = βvₖ + ∇L(θₖ)
θₖ₊₁ = θₖ − αvₖ₊₁
Consistent directions reinforce each other and oscillating directions cancel. On a quadratic, momentum improves the condition-number dependence from κ to roughly √κ, which is the difference between 10⁶ iterations and 10³.
Nesterov's accelerated gradient evaluates the gradient at the point you are about to reach:
vₖ₊₁ = βvₖ + ∇L(θₖ − αβvₖ)
θₖ₊₁ = θₖ − αvₖ₊₁
The lookahead corrects the course before the overshoot happens. The payoff is a rate of O(1/k²) for smooth convex functions, which is provably optimal for any first-order method on that class — you cannot do better with gradients alone [1].
Step 8: Adaptive Methods — RMSprop and Adam
A single α for every parameter is a compromise, and in a neural network it is a bad one. Embedding gradients and bias gradients live on different scales, and a step that is right for one is wrong for the other.
RMSprop divides by a running average of gradient magnitudes:
vₖ = βvₖ₋₁ + (1 − β) ∇L(θₖ)²
θₖ₊₁ = θₖ − α ∇L(θₖ) / √(vₖ + ε)
Adam adds momentum on top and corrects the initialisation bias:
mₖ = β₁mₖ₋₁ + (1 − β₁)∇L(θₖ)
vₖ = β₂vₖ₋₁ + (1 − β₂)∇L(θₖ)²
m̂ₖ = mₖ / (1 − β₁ᵏ), v̂ₖ = vₖ / (1 − β₂ᵏ)
θₖ₊₁ = θₖ − α m̂ₖ / (√v̂ₖ + ε)
The bias correction matters at the start. With β₂ = 0.999, the raw vₖ after one step is 0.001 × g², so √vₖ is about 30× too small and the first steps are enormous. Dividing by 1 − β₂ᵏ fixes exactly that.
The behaviour is per-parameter rescaling: a parameter with consistently large gradients gets a small effective step, and one with tiny gradients gets a large one. That is why Adam is the default — it removes most of the scale problem without you having to think about it [2].
The caveat everyone skips
Adam is not guaranteed to converge. Reddi, Kale, and Kumar built explicit convex problems where Adam's running average of squared gradients decays faster than the true curvature, so the effective step stays large and the iterate never settles [3]. AMSGrad fixes it by keeping the running maximum of v instead of the average. In practice Adam usually works; the point is that "it has an adaptive learning rate" is not a convergence proof.
Step 9: Stochastic Gradient Descent
The full gradient costs O(n) per step:
∇L(θ) = (1/n) ∑ ∇Lᵢ(θ)
SGD replaces it with one example, or a mini-batch:
g = (1/B) ∑_{i ∈ batch} ∇Lᵢ(θ), θ ← θ − αg
The estimator is unbiased, E[g] = ∇L(θ), but it is noisy. That noise is not purely a cost:
- It is cheap.
O(B)per step instead ofO(n). - It enables streaming, because you never need the whole dataset in memory.
- It perturbs the iterate, which helps it leave regions where the gradient is nearly zero but the loss is high.
Variance falls as 1/B while compute rises as B. That trade-off is why batch size is a hardware decision as much as a statistical one.
Large batches need a different learning rate
The linear scaling rule says to multiply α by B/B₀ when you multiply the batch size, with a warmup period at the start so the early, noisy estimates do not blow up the run [5]. Skip the scaling and a larger batch looks like a smaller learning rate — same number of examples per unit of progress, but far fewer updates.
Step 10: A Debugging Checklist
Loss exploding?
αis above2/L. Halve it and check again.- The loss scale changed, so the effective step changed. Recompute the gradient norm.
- Clip the gradient norm at 1.0 as a stopgap, and find the real cause later.
Loss not decreasing?
αis far below1/L. Double it.- Vanishing gradients — check the activation functions and the depth.
- Dead ReLU units — count how many output exactly zero.
- A bug in the gradient. Compare it to a finite-difference estimate.
Oscillating?
αis too large for the steepest direction. That is a condition-number problem, not a tuning problem.- Add momentum, or a schedule that decays
α.
Stuck at a high loss?
- A plateau or a saddle. Momentum and noise help; a different initialisation helps more.
- The model may simply be too small.
Different layers needing different rates?
- That is what Adam is for. Or set layer-wise rates: low for pretrained weights, high for new heads.
Common Pitfalls
- Thinking
αchanges the direction. It does not. It only scales the step along−∇L. - Saying "small enough" without a number. For an
L-smooth function the bound isα < 2/L, andα ≤ 1/Lgives the clean proof. - Forgetting that the loss scale sets the gradient scale. Rescale
yand your learning rate is a different learning rate. - Blaming local minima for slow training. In high dimensions the usual culprits are ill-conditioning and plateaus. Strict saddle points are escaped by gradient descent almost surely [4] — they are not the wall you are hitting.
- Assuming Adam converges because it is popular. It has known convex counterexamples; AMSGrad exists for a reason.
- Scaling the batch without scaling the rate. The linear rule plus warmup is not optional at large batch sizes.
- Trusting a gradient you never checked. A finite-difference comparison costs seconds and catches sign errors, missing
1/nfactors, and mis-broadcast shapes. - Expecting convexity to save a bad step size. Convexity guarantees the landscape is friendly; it says nothing about whether your
αcan walk it.
One more, because it is the most common of all: reporting that a model "does not converge" when the loss was simply multiplied by a constant somewhere in the pipeline. Check the scale before you check the optimizer.
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Convex Optimization | the standard reference for descent methods, the descent lemma, line searches, and the convergence proofs sketched in this article. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read chapter 9 on unconstrained minimisation, then chapter 11 if you want the constrained case. The authors publish the PDF free, and the notation is the one every other optimisation paper assumes you already know.
Wrapping Up
Gradient descent is four facts stacked on top of each other:
- Direction. The directional derivative is
‖∇L‖ cos φ, so the steepest descent direction is−∇L/‖∇L‖. - Step.
αscales that direction, and the Taylor expansion says the loss falls forα < 2/L. - Rate.
O(1/k)for convex, linear for strongly convex, with the condition numberκ = L/μsetting the constant. - Acceleration. Momentum accumulates velocity, Nesterov looks ahead, Adam rescales per parameter, and SGD trades variance for cheap steps.
The update rule packs all of that into one line, and every failure mode you will meet is one of those four facts being violated. When the loss curve misbehaves, do not just change the learning rate — ask which assumption broke. Is the gradient correct? Is the problem ill-conditioned? Is the estimate too noisy?
The likelihood half of this story — why the loss is a negative log-likelihood in the first place — is in maximum likelihood estimation explained, and the reason a penalty term is a prior is in MLE vs MAP. If you want to see the same penalties compared on real data, ridge vs lasso regression has the numbers.
Now go descend some gradients. And check your Taylor expansion.
Frequently Asked Questions
Why does subtracting the gradient decrease the loss?
A first-order Taylor expansion gives L(θ − α∇L) = L(θ) − α‖∇L‖² + (α²/2)∇LᵀH∇L + O(α³). The first-order term is negative for any positive α, so for a small enough step the loss falls. The precise condition for a smooth function is α < 2/L, where L is the gradient's Lipschitz constant.
Does the learning rate change the direction of the update?
No. In plain gradient descent the direction is always the negative gradient; α only scales how far along that line you move. Momentum and Adam change the direction because they use more than the current gradient. This is why tuning α cannot fix a badly conditioned problem, but a preconditioner can.
Why is the negative gradient the steepest descent direction?
The directional derivative in a unit direction v is the gradient dotted with v, which equals the gradient norm times the cosine of the angle between them. That cosine is most negative at 180°, so the direction opposite the gradient gives the fastest local decrease, at a rate equal to the gradient norm.
What learning rate guarantees that gradient descent converges?
For a function whose gradient is L-Lipschitz, any step below 2/L decreases the loss, and a step at or below 1/L gives a clean convergence proof. On a quadratic with largest eigenvalue λ_max, the hard limit is 2/λ_max. Beyond that the iteration diverges even though the function is convex.
Why does Adam sometimes fail to converge?
Adam's update uses a running average of squared gradients, and that average can shrink faster than the true curvature in some problems, producing steps that are too large. Reddi and coauthors built explicit convex counterexamples. AMSGrad fixes it by keeping the running maximum of the second moment instead of the average.
Sources
[1]Bottou, Curtis, and Nocedal, Optimization Methods for Large-Scale Machine Learning (2018) — descent directions, condition numbers, convergence rates, and the optimality of accelerated methods: arxiv.org/abs/1606.04838[2]Kingma and Ba, Adam: A Method for Stochastic Optimization (2015) — the algorithm, the bias correction, and the default hyperparameters: arxiv.org/abs/1412.6980[3]Reddi, Kale, and Kumar, On the Convergence of Adam and Beyond (2018) — the convex counterexamples and the AMSGrad fix: arxiv.org/abs/1904.09237[4]Lee, Simchowitz, Jordan, and Recht, Gradient Descent Only Converges to Minimizers (2016) — why strict saddle points are not the obstacle people think: arxiv.org/abs/1602.04915[5]Goyal et al., Accurate, Large Minibatch SGD (2017) — the linear scaling rule and warmup for large batches: arxiv.org/abs/1706.02677
