Sam Austinon November 12, 202612 min read

Why Lasso Gives Sparse Solutions: A Geometric Explanation

Why Lasso Gives Sparse Solutions: A Geometric Explanation
Contents

Figure 1: Why Lasso Gives Sparse Solutions: A Geometric Explanation — a diamond with corners on the axes, and an ellipse that wants to touch one

You are staring at a regression problem with fifty predictors, and you know only five actually matter. Ridge regression shrinks everything but keeps all fifty. Lasso zeros out the noise. The reason is not a clever algorithm — it is the shape of the constraint region.

I spent too many late nights in grad school on this exact question. The algebra checked out, but the intuition would not stick. Then a professor drew a diamond and a circle on a whiteboard, and everything clicked.

The Core Idea: Constraints Shape Solutions

Optimization is about where you allow the solution to live. Lasso and ridge minimise the same loss — residual sum of squares — and differ only in the region they permit the coefficients to occupy.

  • Ridge constrains the L2 norm: β₁² + β₂² ≤ t, a circle (a sphere in three dimensions, a hypersphere in general).
  • Lasso constrains the L1 norm: |β₁| + |β₂| ≤ t, a diamond (an octahedron in 3D, a cross-polytope in general).

That geometric difference is the whole ballgame. Lasso was introduced by Robert Tibshirani in 1996 [5], and everything else in this article is a consequence of the shape he chose.

The constrained form is equivalent to the penalised form you actually call in code — minimising RSS + λ‖β‖₁ is the same problem as minimising RSS subject to ‖β‖₁ ≤ t, for a matching choice of λ and t. I use both framings because the constraint picture explains why, and the penalty explains how much.

Why Shape Matters

You are minimising RSS subject to a budget on your coefficients. The level sets of RSS are ellipses centred on the ordinary least-squares solution. The constraint region is centred on zero. The solution sits where an expanding ellipse first touches the constraint boundary.

Circles are smooth. Diamonds have corners.

When an expanding ellipse hits a circle, it touches somewhere on the curved edge, and every coefficient stays non-zero. When it hits a diamond, it tends to land on a corner — and corners live on the axes. Axes mean zero coefficients.

This is not a cute visualisation. It is the mechanism. Lasso performs feature selection because of the geometry of its feasible set, not because of anything in the loss function.

Visualizing the Diamond vs. the Circle

Two predictors make this drawable, and two predictors are enough to see the whole effect.

The Ridge Circle

Ridge constrains β₁² + β₂² ≤ t, a circle of radius √t. Smooth, round, no sharp bits anywhere.

The boundary is differentiable everywhere, so at the optimum the gradient of the loss must be exactly antiparallel to the gradient of the constraint. That balance produces proportional shrinkage.

In the special case of an orthonormal design — uncorrelated predictors with unit variance — ridge has the tidy closed form β_ridge = β_ols / (1 + λ). That formula is a special case, not the general one; the general solution is β_ridge = (XᵀX + λI)⁻¹Xᵀy. The orthonormal version is worth remembering because it shows the mechanism plainly: every coefficient is scaled by the same factor, and a scaled non-zero number is still non-zero [2].

Nothing ever hits exactly zero. The shrinkage factor approaches zero as λ grows, but it never arrives.

The Lasso Diamond

Lasso constrains |β₁| + |β₂| ≤ t, a diamond with corners at (±t, 0) and (0, ±t). Four sharp corners sitting directly on the axes.

An expanding ellipse will hit a corner whenever the least-squares solution is not perfectly axis-aligned — which, with real data, is essentially always.

One coefficient goes to zero. Sparsity achieved.

The deeper reason is that the diamond's corners are non-differentiable points. At a corner the penalty has no single gradient, only a set of subgradients, and that set contains the zero vector. A zero in the subgradient set is exactly what lets a coefficient sit at zero while the loss gradient is still non-zero.

There is a second, equivalent way to feel it. Ridge's penalty gradient is proportional to the coefficient, so it fades as the coefficient shrinks. Lasso's penalty gradient is constant in magnitude — it pushes toward zero with the same force regardless of how small the coefficient already is. A constant push can finish the job; a fading push cannot.

Higher Dimensions: The Cross-Polytope

Two dimensions is a drawing. Real problems have p predictors, and the geometry scales without changing character.

The L1 ball is a cross-polytope

In p dimensions, the L1 ball is the convex hull of the 2p points ±t·eᵢ — one for each coordinate axis, in each direction. It has 2p vertices, and every one of them lies on an axis.

  • In 3D that is an octahedron with six vertices.
  • In 100D that is a 200-vertex cross-polytope.
  • Its faces are (p−1)-dimensional simplices; its edges are line segments.

Every vertex is a candidate zero. The L2 ball has none, in any dimension.

Why more dimensions means more sparsity pressure

Two things happen as p grows, and both push toward zeros:

  1. More corners. Each additional predictor adds two vertices, and each vertex is a configuration where all but one coefficient are exactly zero.
  2. Geometry concentrates near the boundary. In high dimensions, most of the volume of a ball sits near its surface, so the expanding ellipse meets the boundary quickly and the vertex-rich structure dominates.

There is also a hard limit worth knowing: when p > n, the lasso can select at most n variables. That is not a quirk of the solver — it follows from the geometry of the solution path, and it is why "lasso will find my 40 true features in 30 samples" is a doomed plan.

The KKT Conditions: Where Math Meets Geometry

If you have taken an optimization course, the KKT conditions are where the picture becomes algebra. They also explain the "sticking" behaviour precisely.

For lasso, the stationarity condition is:

−Xᵀ(y − Xβ) + λz = 0, where z ∈ ∂‖β‖₁

For a single coefficient βⱼ, the subgradient takes three forms:

  • If βⱼ > 0, then zⱼ = 1
  • If βⱼ < 0, then zⱼ = −1
  • If βⱼ = 0, then zⱼ can be any value in [−1, 1]

That third case is the magic. When a coefficient is zero, the subgradient is an interval rather than a point, so the loss gradient can be anything in that range and still satisfy stationarity. The coefficient gets stuck at zero while the data still has a non-zero correlation with it.

For ridge the corresponding term is 2λβⱼ. It equals zero only when βⱼ = 0, and there is no interval, no plateau, no sticking point. The coefficient can pass through zero smoothly on its way to the other side, but it has no reason to stop there.

There is a practical reading of the same condition: a variable can only enter the model when its correlation with the current residual exceeds λ in magnitude. Below that threshold, the coefficient is pinned at zero — which is exactly the soft-thresholding you see in every coordinate-descent implementation.

The Regularization Path: Watching Sparsity Emerge

Plot every coefficient as λ goes from zero to large, and the difference becomes visible without any algebra.

  • Ridge paths are smooth curves that decay toward zero and approach it asymptotically. They never arrive.
  • Lasso paths are piecewise linear — straight segments with kinks where variables enter or leave [4]. They hit zero and stay there.

As λ increases, coefficients drop out one at a time, and the model gets sparser. The count of non-zero coefficients generally falls as the penalty grows, though the active set can change in both directions along the path, so do not treat it as a strict one-way staircase.

This path property is why lasso is so useful for feature selection. You compute the whole sequence of models at once, then let cross-validation pick λ — and the feature set comes with it, for free. That is the same logic behind the wrapper and embedded methods in my feature selection comparison, where lasso sits in the embedded family.

The "Why Not L0?" Question

Why not penalise the number of non-zero coefficients directly?

Because it is NP-hard. Minimising RSS subject to ‖β‖₀ ≤ k means choosing which k of the p predictors to keep. For 50 predictors and 5 true signals that is C(50,5) = 2,118,760 subsets. For a thousand predictors it is a number with no practical meaning.

L1 is the convex relaxation of L0. The L1 ball is the convex hull of the L0 unit "ball" — the set of points with at most one non-zero coordinate of magnitude one. Relaxing to the convex hull turns an intractable search into a convex problem with a single global minimum, solvable in polynomial time.

That is the most elegant idea in regularization theory, and it is why lasso is not just a heuristic: it is the tightest convex thing you can do while still asking for zeros [1].

When Lasso Doesn't Give Sparsity

Fair is fair. Lasso encourages zeros; it does not guarantee them for every coefficient in every dataset.

  1. Orthogonal design with weak signals. If predictors are uncorrelated and the true coefficients are tiny, the ellipse can land on a flat face rather than a vertex. You get shrinkage without zeros.
  2. Highly correlated predictors. Lasso picks one from a group and zeros the rest — and which one survives is unstable across samples. This is the "bet on one horse" problem, and it is why elastic net exists.
  3. p > n. As above, at most n variables can be selected, no matter how many true signals exist.
  4. The irrepresentable condition fails. When noise variables are correlated with signal variables in the wrong way, lasso cannot separate them — it either misses real signals or admits noise [1].

Know the assumptions or you will misread the zeros. A zero coefficient means "not selected", not "proven irrelevant".

Practical Takeaways

  • Use lasso when you believe many coefficients are exactly zero, when selection matters more than prediction stability, when you want an interpretable short list, or when p is large relative to n.
  • Use ridge when signals are dense, when multicollinearity is the main headache, when prediction accuracy matters more than interpretation, or when you need stable coefficients across resamples.
  • Use elastic net when predictors are correlated and you want grouping behaviour, when the lasso path is too unstable, or when you cannot choose between the two — it blends α‖β‖₁ + (1−α)‖β‖₂.

Standardise your predictors first. The L1 penalty is scale-dependent, so a variable measured in millimetres gets penalised differently from the same variable measured in metres. That one habit prevents most "lasso selected nonsense" complaints, and scikit-learn's preprocessing guide covers the mechanics [3].

A Personal Note on Teaching This

That whiteboard moment came down to one sentence: "Ridge shrinks. Lasso selects. The diamond has corners. The circle doesn't. That's it."

It is not the full mathematics — subgradients, KKT conditions, coordinate-descent convergence, irrepresentable conditions, asymptotic consistency — there is a PhD thesis in the details. But the geometry is the mental model that survives, and it is the one I still reach for when a model does something unexpected.

I have watched PhD candidates grind through the algebra for weeks. One diagram, thirty seconds, and the intuition lands. Never underestimate a good picture.

Common Pitfalls

  • Reading a zero as "no effect". A zero means "not selected given the others", which is a different claim.
  • Expecting zeros without standardising. Scale changes the penalty, so unscaled features produce arbitrary selections.
  • Tuning λ on the test set. Use cross-validation on training data; the test set is for the final number.
  • Assuming the lasso found the true features. With correlated groups it finds a set, not the set.
  • Using lasso when p > n and expecting everything. At most n variables survive.
  • Treating sparsity as interpretability. A short list of unstable coefficients is not an explanation.
  • Blaming the solver for non-zeros. Coordinate descent converges to exact zeros when they exist; tiny non-zeros usually mean λ is too small.
  • Forgetting the geometry when it fails. If the contour lands on a face, you get shrinkage, not selection — and no amount of tuning changes the shape.

Best Book to Read

CoverBookDescriptionGet it
Cover of “Statistical Learning with Sparsity: The Lasso and Generalizations” Statistical Learning with Sparsity: The Lasso and Generalizationsby Trevor Hastie, Robert Tibshirani, and Martin Wainwright the definitive treatment of the lasso, its KKT conditions, the irrepresentable condition, and the convex-relaxation argument behind L1. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Read chapter 2 for the geometry and chapter 11 for the conditions under which selection succeeds. It is the book this article is a summary of, and the free PDF is on the authors' site.

Wrapping Up

Lasso gives sparse solutions because its L1 constraint region is a cross-polytope with vertices on every axis. Expanding RSS ellipses tend to touch corners, corners sit where coefficients equal zero, and the subgradient interval at a corner is what keeps them there.

Ridge's circular constraint has no corners, so no coefficient ever lands exactly on zero. Same loss, different shape, completely different behaviour.

Is that the whole story? No. But for intuition that sticks, the diamond beats the equations every time. If you want the wider map of which mathematics actually matters for ML, I wrote that up in what practitioners really need.

Now go build something sparse. And maybe skip the all-nighter.

Frequently Asked Questions

Why does L1 regularization produce zero coefficients?

The L1 constraint region is a diamond with sharp corners that sit on the coordinate axes. Residual contours expand from the least-squares solution and tend to touch the region at a corner, where at least one coefficient is exactly zero. The L2 region is a smooth circle with no corners, so contact lands on a curved edge where every coefficient stays non-zero.

Is lasso always sparse?

No. Lasso encourages sparsity but does not guarantee it. With orthogonal predictors and weak signals, contours can land on a flat face and give shrinkage without zeros. Correlated groups produce one arbitrary survivor, and when p is greater than n the lasso can select at most n variables.

Why not use an L0 penalty instead of L1?

L0 gives exactly the sparsity you asked for, but minimising it requires searching subsets, which is NP-hard. For 50 predictors and 5 true signals that is 2,118,760 combinations. L1 is the tightest convex relaxation of L0, so it has one global minimum and solves in polynomial time.

Does ridge regression ever produce exact zeros?

Practically never. Ridge shrinks every coefficient proportionally, and its penalty gradient fades as coefficients approach zero, so they get arbitrarily small without ever landing on zero. Exact zeros in ridge output come from numerical underflow, not from the method.

What should I use when predictors are highly correlated?

Elastic net. Lasso picks one variable from a correlated group and zeroes the rest, and which one survives is unstable across samples. Elastic net adds an L2 term, which spreads weight across the group and stabilises selection.

Sources

What are You Looking For?

esc