Ridge vs Lasso Regression: Regularization Explained

Contents

Figure 1: Two penalties, one goal — keep the model honest

Two penalties, one tuning knob — and one of them does your feature selection for you

Give an ordinary linear regression too many features and too little data, and it will happily fit the noise. Its coefficients swing wildly, two correlated features cancel each other out with huge opposite weights, and the model performs beautifully on training data and badly on anything new. Ridge and Lasso are the two classic fixes, and the difference between them is small in the formula and large in behavior: Ridge shrinks coefficients, Lasso shrinks them and sets some exactly to zero.

Regularization is the standard cure: add a penalty for large coefficients so the model prefers simpler explanations. Below are the formulas, the geometry that explains the zeros, three runnable scikit-learn demos, and a full comparison on realistic data.

The Problem Regularization Solves

Ordinary least squares (OLS) chooses the coefficients that minimize the sum of squared errors — the objective the linear regression guide in this series derives from scratch. With many features, strongly correlated features, or fewer samples than features, it goes wrong in specific ways:

• High variance: small changes in the training data produce very different coefficients.
• Unstable coefficients: with correlated features, OLS can assign +50 to one and −49 to its twin, a combination that fits the training data and generalizes poorly.
• No unique solution when features outnumber samples.

Regularization accepts a little bias in exchange for a large drop in variance, which usually lowers error on new data. That's the bias-variance tradeoff from the algorithms guide in this series, applied to linear models: shrinkage pulls coefficients toward zero (more bias), and the model stops chasing noise (less variance).

The Two Penalties

Both methods add a penalty term to the least-squares objective, weighted by a tuning parameter alpha (α) that controls how strongly to penalize.

Ridge (L2 penalty) penalizes the sum of squared coefficients:

minimize: RSS + alpha * sum(w_j**2)

Lasso (L1 penalty) penalizes the sum of absolute values:

minimize: RSS + alpha * sum(abs(w_j))

With alpha = 0 you recover OLS, and as alpha grows, coefficients shrink toward zero. The intercept isn't penalized.

One practical warning: scikit-learn scales the objectives differently — Lasso's squared-error term is divided by 2n — so the same alpha value means different things in Ridge and Lasso. Never compare alphas across the two models, only tune each one separately.

Why L1 Makes Zeros and L2 Doesn't

The geometric picture is the classic explanation. Imagine two coefficients. Minimizing the error subject to a penalty budget is equivalent to finding where the error contours first touch the allowed region:

• Ridge: the region is a circle (a sphere in higher dimensions). The contours typically touch it at a point where both coefficients are small but nonzero.
• Lasso: the region is a diamond with sharp corners on the axes. The contours are far more likely to hit a corner, where one coefficient is exactly zero.

The corners are the whole story. A contour hitting a flat side lands somewhere in the middle of an edge — nonzero coordinates. A contour hitting a corner lands on an axis — one coordinate exactly zero.

Equivalently, ridge's penalty gradient shrinks in proportion to the coefficient, so it fades as coefficients get small and never quite reaches zero, while lasso applies a constant push toward zero regardless of size, which can finish the job. In Bayesian terms, ridge corresponds to a Gaussian prior on the coefficients and lasso to a Laplace prior, which is sharply peaked at zero.

Behavior Side by Side

Ridge:

• Shrinks all coefficients smoothly but keeps every feature.
• Handles correlated features gracefully by spreading weight across them.
• Works when many features each contribute a little.
• Has a stable, closed-form solution and is fast.

Lasso:

• Produces a sparse model, with many coefficients exactly zero, so it doubles as feature selection.
• With a group of correlated features, tends to pick one arbitrarily and drop the others, which can be unstable across resamples.
• Can select at most as many features as there are samples when features outnumber samples.
• Gives a more interpretable model when the true signal is sparse.

Elastic Net blends both penalties with a mixing parameter, l1_ratio. It keeps lasso's sparsity while behaving more stably with correlated groups, which makes it a strong default when you have many correlated features and still want selection.

Correlated Features, Demonstrated

Here's the clearest way to see the difference. Two features are nearly copies of each other, and only one truly matters:

import numpy as np
from sklearn.linear_model import Lasso, LinearRegression, Ridge

rng = np.random.default_rng(0)
x1 = rng.normal(size=200)
x2 = x1 + rng.normal(scale=0.05, size=200)          # almost a copy of x1
X = np.column_stack([x1, x2, rng.normal(size=(200, 3))])
y = 3 * x1 + rng.normal(scale=1.0, size=200)

for name, model in [("OLS", LinearRegression()),
                    ("Ridge", Ridge(alpha=10)),
                    ("Lasso", Lasso(alpha=0.1))]:
    model.fit(X, y)
    print(f"{name:<6}", np.round(model.coef_, 2))

Expect OLS to produce erratic values for the first two coefficients, ridge to split the weight roughly evenly between them, and lasso to keep one and zero the other. (Features here are already on the same scale, so I skipped standardization in this tiny demo. In real projects, always standardize.) Run it with different seeds and watch which twin lasso picks — that arbitrariness is a real property of lasso, not a bug.

A Full Comparison on Realistic Data

This script compares OLS, Ridge, Lasso, and Elastic Net on a dataset with 60 features, of which only 6 matter, and uses cross-validation to choose each alpha:

import numpy as np
import pandas as pd
from sklearn.datasets import make_regression
from sklearn.linear_model import ElasticNetCV, LassoCV, LinearRegression, RidgeCV
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y, true_coef = make_regression(
    n_samples=120, n_features=60, n_informative=6,
    noise=15, coef=True, random_state=0,
)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)

cv = KFold(5, shuffle=True, random_state=0)
models = {
    "OLS": make_pipeline(StandardScaler(), LinearRegression()),
    "Ridge": make_pipeline(StandardScaler(),
                           RidgeCV(alphas=np.logspace(-3, 3, 50), cv=cv)),
    "Lasso": make_pipeline(StandardScaler(),
                           LassoCV(cv=cv, random_state=0, max_iter=20000)),
    "ElasticNet": make_pipeline(StandardScaler(),
                                ElasticNetCV(l1_ratio=[0.2, 0.5, 0.8, 0.95], cv=cv,
                                             random_state=0, max_iter=20000)),
}

rows = []
for name, model in models.items():
    model.fit(X_train, y_train)
    coef = model[-1].coef_
    rmse = mean_squared_error(y_test, model.predict(X_test)) ** 0.5
    rows.append({
        "model": name,
        "test_RMSE": round(rmse, 2),
        "nonzero_coefs": int((np.abs(coef) > 1e-8).sum()),
        "alpha": getattr(model[-1], "alpha_", None),
    })
print(pd.DataFrame(rows))

lasso_coef = models["Lasso"][-1].coef_
kept = int(((lasso_coef != 0) & (true_coef != 0)).sum())
print(f"true informative features kept by Lasso: {kept} of {int((true_coef != 0).sum())}")

Typically you'll see OLS overfit badly with 60 features and only 72 training samples, ridge and lasso both improving test error, and lasso keeping far fewer nonzero coefficients. Your exact numbers will vary, but the pattern is the lesson.

Notice the structure: scaling lives inside the pipeline so it's fit on each training fold only, and the cross-validated estimators (RidgeCV, LassoCV, ElasticNetCV) choose alpha without touching the test set.

Visualizing the Regularization Paths

A coefficient path plots every coefficient as alpha changes, and it makes the two behaviors visible at a glance:

import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge, lasso_path

Xs = StandardScaler().fit_transform(X_train)

ridge_alphas = np.logspace(-2, 4, 60)
ridge_coefs = np.array([Ridge(alpha=a).fit(Xs, y_train).coef_ for a in ridge_alphas])
lasso_alphas, lasso_coefs, _ = lasso_path(Xs, y_train, alphas=np.logspace(-2, 2, 60))

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(np.log10(ridge_alphas), ridge_coefs)
axes[0].set_title("Ridge: coefficients shrink smoothly")
axes[1].plot(np.log10(lasso_alphas), lasso_coefs.T)
axes[1].set_title("Lasso: coefficients hit exactly zero")
for ax in axes:
    ax.set_xlabel("log10(alpha)"); ax.set_ylabel("coefficient")
plt.tight_layout(); plt.show()

In the ridge panel, lines glide toward zero without touching it. In the lasso panel, lines drop to zero one by one as alpha rises, and the order in which they die tells you roughly which features are most robustly useful.

Choosing Alpha

Use cross-validation, and use a logarithmic grid since alpha spans orders of magnitude. RidgeCV, LassoCV, and ElasticNetCV handle this efficiently, with LassoCV using the regularization path to avoid refitting from scratch.

A useful refinement is the one-standard-error rule: pick the largest alpha whose CV score is within one standard error of the best. That gives a simpler, more heavily regularized model with statistically indistinguishable accuracy — the same idea used for tree pruning in the decision trees guide. For time-ordered data, use time-aware cross-validation instead of shuffled folds.

Always Standardize First

The penalty treats all coefficients equally, so features on different scales get penalized unfairly: a feature measured in thousands needs a tiny coefficient and is barely penalized, while one measured in decimals needs a large coefficient and is penalized heavily.

Standardize features (zero mean, unit variance) before fitting, inside a pipeline so the scaling is learned only from training data. After scaling, coefficients are comparable across features, which makes importance rankings meaningful.

Which Should You Use?

• Use ridge when you believe many features contribute small effects, features are strongly correlated, or you care mostly about prediction accuracy and don't need feature selection.
• Use lasso when you suspect only a few features matter, want automatic feature selection, or need a sparse, interpretable model — especially with many more features than samples.
• Use elastic net when you want sparsity but have correlated groups of features, which is common in practice. It's often the safest default when you can't decide.
• Use plain OLS when you have plenty of data, few features, and little collinearity. Regularization helps less there.
• Consider trees or boosting when relationships are non-linear, since regularized linear models are still linear, as the algorithms guide explains.

Caveats Worth Knowing

• Lasso selection can be unstable. Which feature gets picked among correlated ones can change with small data perturbations. If selection itself matters, check stability by refitting on bootstrap resamples and seeing how consistently each feature is chosen.
• Penalized coefficients are biased. Shrinkage pulls even true coefficients toward zero. A common remedy is a relaxed or post-lasso refit: use lasso to select features, then refit an unpenalized or lightly penalized model on the survivors, ideally evaluated on separate data.
• Don't report ordinary p-values after lasso selection. Choosing features with the same data you test on invalidates standard inference, and specialized methods (such as debiased lasso or sample splitting) exist for valid intervals.
• Coefficients aren't causal effects. Regularized models describe predictive association, nothing more.
• It's more than regression. The same penalties apply to classification: logistic regression uses L2 by default, and in recent scikit-learn versions you control the penalty type with l1_ratio (the older penalty argument is deprecated), with C acting as the inverse of regularization strength, as covered in the logistic regression guide. Neural network "weight decay" is essentially an L2 penalty too, though adaptive optimizers like Adam treat it differently, which is why decoupled variants like AdamW exist.

Common Pitfalls

• Skipping standardization, so penalties land unevenly across features.
• Comparing alpha values between Ridge and Lasso, when scikit-learn scales their objectives differently.
• Picking alpha by test-set performance instead of cross-validation.
• Scaling the whole dataset before cross-validation, which leaks information.
• Trusting lasso's chosen features as "the" important ones when correlated alternatives exist.
• Reporting p-values or confidence intervals from a lasso-selected model without correction.
• Ignoring convergence warnings from Lasso. Standardize features, increase max_iter, or loosen tol.
• Using shuffled cross-validation on time series.
• Expecting regularization to fix non-linearity or leakage. It won't.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the difference between Ridge and Lasso?

Both add a penalty on coefficient size to the least-squares objective, weighted by alpha. Ridge uses the L2 penalty (sum of squared coefficients) and shrinks every coefficient smoothly toward zero while keeping all features; Lasso uses the L1 penalty (sum of absolute values) and drives some coefficients to exactly zero, giving you a sparse model with built-in feature selection. At alpha = 0 both reduce to ordinary least squares.

Why does Lasso produce zeros and Ridge doesn't?

Geometry: the L1 penalty budget is a diamond whose corners sit on the axes, so the error contours usually touch a corner where one coefficient is exactly zero; the L2 budget is a circle with no corners, so contact points are generically nonzero. Equivalently, L1 applies a constant push toward zero while L2's push shrinks with the coefficient — in Bayesian terms, a Laplace prior (peaked at zero) versus a Gaussian prior.

Should I standardize features before regularizing?

Yes. The penalty treats every coefficient equally, so a feature measured in thousands keeps a tiny coefficient and barely gets penalized while a decimal-scaled feature is penalized heavily. Standardize to zero mean and unit variance inside a pipeline so the scaling is learned only from training folds — as a bonus, coefficients become comparable, so importance rankings are meaningful.

How do I choose alpha for Ridge or Lasso?

Cross-validation on a logarithmic grid, since alpha spans orders of magnitude. RidgeCV, LassoCV, and ElasticNetCV do this efficiently, and the one-standard-error rule — the largest alpha within one standard error of the best — gives a simpler, more regularized model with statistically indistinguishable accuracy. Never compare alpha values across Ridge and Lasso: scikit-learn scales their objectives differently.

Wrapping This Up

Ridge and Lasso both tame overfitting by penalizing large coefficients. Ridge uses the squared (L2) penalty, shrinking everything smoothly and sharing weight among correlated features. Lasso uses the absolute (L1) penalty, which has corners that drive coefficients to exactly zero and give you built-in feature selection, at the cost of some instability with correlated predictors. Elastic net blends the two.

Which one wins? On dense, correlated data ridge often predicts best, on sparse signals lasso is easier to interpret, and elastic net is a dependable compromise. Standardize your features, choose alpha with cross-validation on a log grid — and run the comparison above on your own data, plot the paths, and let the cross-validated results settle it.

Related Articles

• Linear Regression in Python: From Scratch to Scikit-Learn
• Bias-Variance Tradeoff Explained with Visual Examples
• Logistic Regression Explained: Build a Classifier Step by Step
• Decision Trees in Python: Build, Visualize, and Prune