Contents
Figure 1: MLE vs MAP: How Priors Turn into Regularization — one extra term in the objective, and the same solver becomes a Bayesian estimator
You fit a logistic regression and the coefficients come back enormous. Standard errors are huge, training accuracy is perfect, and the model falls apart on held-out data. Then someone says: "add a prior." The coefficients shrink, the model generalises, and it looks like a different technique.
It is not a different technique. It is the same optimisation with one extra term, and that term has a name in statistics: a prior.
The translation runs both ways. Your ridge regression is a Gaussian prior. Your lasso is a Laplace prior. The alpha you tune is not the prior precision — it is the noise variance multiplied by the prior precision, which is why the same number means different things on different datasets. Two languages, one piece of arithmetic.
The Core Difference: Point Estimate vs Posterior Mode
MLE asks which parameter makes the data most likely. MAP asks which parameter is most probable once you combine the data with what you believed beforehand.
θ̂_MLE = argmax_θ L(θ | data) = argmax_θ P(data | θ)
θ̂_MAP = argmax_θ P(θ | data) = argmax_θ [ P(data | θ) × P(θ) ] / P(data)
The denominator does not contain θ, so it cannot move the argmax. Drop it, take logs, and the whole method collapses to one line:
θ̂_MAP = argmax_θ [ log P(data | θ) + log P(θ) ]
Log-likelihood plus log-prior. That is the entire difference, and everything else in this article is a consequence of what you choose for the second term.
Note what MAP is not. It is a point estimate, not a distribution, and it is not invariant to how you parameterise the model — a point I come back to in the pitfalls, because it is the strongest objection to using MAP at all.
Gaussian Prior = Ridge Regression
Start with the linear model y = Xβ + ε with ε ~ N(0, σ²I). The log-likelihood, up to constants, is:
log P(y | β) = −‖y − Xβ‖² / (2σ²)
Now put a zero-mean Gaussian prior on the coefficients, β ~ N(0, τ²I):
log P(β) = −‖β‖² / (2τ²) + const
Add them and maximise. Maximising the sum is the same as minimising its negative:
‖y − Xβ‖² / (2σ²) + ‖β‖² / (2τ²)
Multiply through by 2σ² and you get ridge regression exactly:
‖y − Xβ‖² + (σ²/τ²) ‖β‖²
So λ = σ²/τ². The prior variance τ² controls the shrinkage: a small τ² is a confident prior, which means a large λ and heavy shrinkage. A large τ² is a vague prior, which means a small λ and an answer close to least squares.
This is why ridge works. The likelihood pulls the estimate toward the least-squares solution; the prior pulls it toward zero; MAP finds the point where the two forces balance.
One precision worth keeping: this equivalence treats σ² as fixed. If you estimate the noise variance jointly with the coefficients, the penalty is no longer exactly ridge — integrating out σ² under an inverse-gamma prior gives a multivariate-t marginal and a penalty that changes with the residuals.
Laplace Prior = Lasso
Now swap the Gaussian for a Laplace, also called a double-exponential, prior:
P(β) ∝ exp(−|β| / b)
Its log is −‖β‖₁ / b plus a constant, so the MAP objective becomes:
‖y − Xβ‖² / (2σ²) + ‖β‖₁ / b
Multiply by 2σ² again and you have lasso with λ = 2σ²/b.
The mechanism is the shape of the density. A Gaussian density is smooth at zero, so its log is smooth, so the penalty nudges small coefficients without ever finishing them off. A Laplace density has a sharp peak at zero, which means its log has a kink there, which means the penalty pushes with constant force regardless of how small the coefficient already is. That kink is the diamond, and the diamond is why lasso produces exact zeros — the geometry I worked through in why lasso gives sparse solutions.
There is a subtlety the "Bayesian lasso" literature makes clear. Park and Casella's Bayesian treatment of the lasso uses the posterior mean, not the mode, and the posterior mean under a Laplace prior is not sparse [2]. The zeros you get from sklearn come from the mode. If you sample the posterior and average, the coefficients are small but non-zero — which is the honest answer, because the posterior genuinely has mass on both sides of zero.
Why the Prior Shape Is the Regularization Geometry
Every penalty is a log-prior, so every regulariser has a shape, and the shape is the behaviour.
Prior on β |
Log-prior | Penalty | Behaviour |
|---|---|---|---|
Gaussian N(0, τ²I) |
−‖β‖₂²/(2τ²) |
L2 | smooth shrinkage, never zero |
| Laplace | −‖β‖₁/b |
L1 | kink at zero, sparse |
| Product of the two | sum of the two | elastic net | sparse plus grouping |
This is the translation layer in one table. When you pick L1 over L2, you are not picking an algorithm — you are asserting that you believe most coefficients are exactly zero. That is a claim about the world, and it can be wrong.
The Elastic Net: A Product of Priors, Not a Mixture
The elastic net penalty is α‖β‖₁ + (1−α)‖β‖₂², and the standard story is that it "mixes" a Laplace and a Gaussian prior. That is the wrong word, and the distinction matters.
Penalties add, and adding penalties means multiplying densities. The elastic net corresponds to a prior whose density is proportional to the product of a Gaussian and a Laplace:
P(β) ∝ exp(−β²/(2τ²)) × exp(−|β|/b)
so the log-prior is the sum of the two terms. It is one prior with two features, not a random draw from one of two priors. A genuine mixture would put a spike of probability on "this coefficient is exactly zero" and behave differently again.
The Laplace distribution itself is a Gaussian scale mixture: draw a variance from an exponential distribution, then draw β from a normal with that variance, and the marginal is Laplace. The exponential mixing density concentrates mass near zero variance, which is where the spike comes from. That construction is the seed of the hierarchical priors later in this article.
What λ Actually Is
The claim that λ is the prior precision is close but not correct. From the derivation, λ = σ²/τ², so:
λ = (noise variance) × (prior precision)
That extra factor explains a practical annoyance. σ² carries the units of y squared, so λ is not scale-free. Rescale your target — dollars to thousands of dollars — and the λ that produces the same shrinkage changes by a factor of a million. This is why copying alpha=1.0 from a tutorial onto a new dataset is meaningless, and why sklearn's regularisation strength interacts with whatever preprocessing you did.
It also explains the intuition people usually get backwards. Noisier data means a larger σ², which means a larger λ for the same prior, which means more shrinkage. That is correct behaviour: when the likelihood is unreliable, the prior should carry more weight.
Automatic Relevance Determination
Instead of one shared variance, give every coefficient its own:
βⱼ ~ N(0, τⱼ²)
Then learn the τⱼ by maximising the marginal likelihood — the evidence — rather than fixing them. Irrelevant features drive their own τⱼ toward zero, which shrinks their coefficients to nothing and performs selection as a side effect. This is automatic relevance determination, and it is the ancestor of the modern sparse Bayesian methods.
A common mislabelling: ARD is often called "the Bayesian version of the group lasso." It is not. Group lasso handles coefficients that belong to known groups and should be selected together. ARD is closer to the adaptive lasso or to iteratively reweighted L1, because the weights are learned per coefficient from the data rather than imposed by a grouping you defined.
Uncertainty: Why MAP Is Not Full Bayes
MAP gives you a point. Full Bayes gives you a distribution, and the distribution is usually the point.
- The posterior mean and the MAP coincide only when the posterior is symmetric. Skewed posteriors — which is most posteriors near a boundary — put them in different places.
- The posterior variance is the uncertainty, and it is the thing you actually need for a credible interval.
- A credible interval has the interpretation people wrongly give to a confidence interval: a 95% credible interval contains the parameter with 95% posterior probability, given the model and the prior.
If you need uncertainty, do not stop at MAP. Optimise to find the mode, then either sample the posterior or use a Laplace approximation — a Gaussian fitted at the mode with covariance equal to the inverse Hessian. The Laplace approximation is fast and often good enough, but it is an approximation to a shape you have not checked.
The Flat Prior Limit: MAP → MLE
Push the prior variance to infinity and the log-prior becomes a constant.
- Gaussian prior,
τ² → ∞: the penaltyβ²/(2τ²)vanishes, and MAP is least squares. - Laplace prior,
b → ∞: the penalty|β|/bvanishes, and MAP is least squares.
MLE is MAP with a flat prior. The frequentist estimator is a special case of the Bayesian one, not a rival to it. That is the basis of the joke that everyone is a Bayesian and some people just refuse to write the prior down.
The limit is worth stating carefully, though. A flat prior over an unbounded parameter is improper, and an improper prior only produces a valid posterior when the likelihood is strong enough to normalise it. With separation in logistic regression, the likelihood is not strong enough — which is exactly why "just use a flat prior" fails in the case that motivated this article.
When MAP Beats MLE (and Vice Versa)
MAP wins when:
nis small andpis large. The prior stabilises an estimate that the data cannot pin down, and in thep > ncase the MLE may not exist at all.- Predictors are collinear. A prior breaks the symmetry that makes the MLE explode in opposite directions for correlated features.
- You have real domain knowledge. If you know effects are small, or sparse, or bounded, encoding that is not cheating — it is using information you have.
- You want a regularisation path. Cross-validated
λis an empirical estimate of how much to trust the prior.
MLE wins when:
nis large andpis small. The likelihood dominates, and the prior stops mattering.- You have no credible prior. A flat prior is honest ignorance; a made-up prior is not.
- You need invariance. MLE is invariant to reparameterisation and MAP is not, so if the parameterisation is arbitrary, MLE is the safer target.
- You want clean asymptotics. The MLE's asymptotic normality gives standard errors, Wald tests, and likelihood-ratio tests without extra assumptions.
Empirical Bayes: Learning the Prior from Data
If you cannot choose the prior, estimate it. Empirical Bayes treats the prior's hyperparameters as unknowns and maximises the marginal likelihood:
P(data | τ²) = ∫ P(data | β) P(β | τ²) dβ
The integral removes β and leaves a function of τ² alone. Maximise that, plug the estimate back into the prior, and the data has chosen its own shrinkage strength. For a Gaussian prior with Gaussian likelihood the integral is closed-form; for a Laplace prior you need an approximation or sampling.
This is what cross-validation approximates. CV picks λ to maximise predictive performance on held-out folds; empirical Bayes picks λ to maximise the evidence. Different objectives, usually similar answers, and CV is the one that needs no model for the prior at all.
A Concrete Example: Logistic Regression with Priors
Binary classification with P(y = 1 | x) = σ(xᵀβ). The MLE maximises
∑ [ yᵢ log pᵢ + (1 − yᵢ) log(1 − pᵢ) ]
Add a Gaussian log-prior and you have ridge-penalised logistic regression. Add a Laplace log-prior and you have lasso-penalised logistic regression, which selects features. Neither changes the solver's job — you still optimise, just a different objective [4].
Separation: the case where the prior is not optional
Complete separation means some linear combination of predictors perfectly predicts the outcome. The likelihood then keeps rising as the coefficients grow, so no finite MLE exists, and the software either warns you or returns enormous coefficients with enormous standard errors.
Any proper prior fixes it, because the log-prior eventually grows faster than the log-likelihood as you move away from zero. Gaussian gives a finite, shrunken answer; Laplace gives a finite, sparse one.
Firth's penalised likelihood is exactly this idea with a specific prior — the Jeffreys prior, P(β) ∝ |I(β)|^{1/2} — which removes the first-order bias as well as taming separation [5]. It is standard practice in epidemiology, and it is a MAP estimate whether or not the paper calls it one.
A defensible default: a Cauchy prior with scale 2.5 on coefficients, applied to standardised predictors [1]. It is weakly informative, it keeps the estimate finite, and it does not pretend to know more than you do. Standardising first matters — the scale of 2.5 is calibrated for predictors on a common scale.
Hierarchical Priors: Global-Local Shrinkage
Why stop at one level of prior? Let the data learn how much to shrink each coefficient:
βⱼ | λⱼ, τ ~ N(0, τ² λⱼ²), with λⱼ ~ Half-Cauchy(0, 1) and τ ~ Half-Cauchy(0, 1)
The global term τ pulls everything toward zero. The local terms λⱼ let large signals escape that pull. A coefficient with a large λⱼ gets a wide prior and is barely penalised; one with a tiny λⱼ is crushed.
The horseshoe prior is the best-known member of this family [3], and its behaviour is easiest to remember as two properties of the Half-Cauchy:
- Heavy tails on
λⱼmean a genuinely large signal pays almost no bias. - A sharp peak at zero means noise coefficients are shrunk hard.
That combination is what plain lasso cannot do: lasso applies the same penalty to a huge signal and a tiny one, so it biases the large coefficients toward zero. Global-local priors were designed to fix that specific failure, and they are the current default for sparse Bayesian regression.
Computational Reality Check
MAP is an optimisation problem, and so is MLE. Same order of cost, same solvers, same convergence diagnostics.
Full Bayes is an integration problem, which means MCMC, variational inference, or Laplace approximation. That is a different order of effort and a different set of failure modes.
For large models — neural networks, huge datasets — MAP is often the practical sweet spot: you get the regularisation benefit, the optimisation speed, and no sampling. The price is that you have a point estimate dressed as a posterior, and any uncertainty you report has to come from somewhere else.
Common Pitfalls
- Calling MAP "the Bayesian answer". It is a point estimate. The Bayesian answer is the posterior, and MAP is one summary of it.
- Assuming MAP is invariant. It is not. Estimate a standard deviation instead of a variance and the MAP estimate moves. The MLE does not.
- Expecting the posterior mean under a Laplace prior to be sparse. It is not. Only the mode is, which is why the Bayesian lasso reports a mean that never quite hits zero.
- Treating
λas the prior precision. It isσ²/τ². Rescaleyand the sameλmeans something different. - Fitting without standardising. The L1 penalty is scale-dependent, and a Cauchy prior's scale is meaningless on unstandardised inputs.
- Believing a flat prior is objective. Flat on
βis not flat on1/β, on a probability, or on a standard deviation. "Objective" priors are a choice too. - Reading a ridge coefficient as an effect size. It is a biased, shrunken estimate, and the bias is the point.
- Using a Laplace approximation without checking the shape. If the posterior is skewed or multimodal, a Gaussian fitted at the mode is a confident lie.
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Statistical Rethinking | a second course in Bayesian modelling that builds from the coin-flip likelihood up to multilevel models, with the prior-as-regulariser idea running through the whole book. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read the chapters on sampling and on multilevel models first. McElreath is unusual in showing the consequences of a prior rather than just the algebra, which is exactly what this topic needs. The lecture videos are free online and match the book chapter for chapter.
Wrapping Up
MAP maximises likelihood plus log-prior, and the log-prior is a penalty. Gaussian gives ridge, Laplace gives lasso, the product of the two gives elastic net, and a hierarchy of priors gives the horseshoe.
Regularisation is not a separate idea from Bayesian inference. It is Bayesian inference stopped at the mode, with a particular prior, and the parameter you tune is the noise variance divided by the prior variance.
Next time you set alpha, remember what you are actually choosing. You are choosing a prior variance and a prior shape — that is, you are making a claim about the world. Make the claim explicit, then defend it, and if you want the uncertainty that a prior implies, do not stop at the mode. The full derivation of the likelihood side of this argument is in maximum likelihood estimation explained, and the practical comparison of the two penalties is in ridge vs lasso regression.
Now go put some priors on your parameters. And maybe admit you are a Bayesian.
Frequently Asked Questions
Is MAP estimation the same thing as regularization?
Yes. Every penalty term you add to a loss function is the negative log of some prior, up to constants. Ridge is a Gaussian prior, lasso is a Laplace prior, and elastic net is the product of the two. Choosing a penalty and choosing a prior are the same act described in two vocabularies.
Why does a Gaussian prior produce L2 and a Laplace prior produce L1?
The penalty is minus the log-prior. The log of a Gaussian density is quadratic, so you get a squared penalty. The log of a Laplace density is the absolute value, so you get an L1 penalty. The sharp peak of the Laplace density at zero is what turns into exact zeros in the estimate.
What is the difference between MAP and full Bayesian inference?
MAP returns a single point, the posterior mode. Full Bayes returns the whole posterior distribution, so you get means, credible intervals, and uncertainty that reflects the data. MAP is cheap because it is just optimisation, but it throws away the shape of the posterior.
Is the MAP estimate invariant to reparameterisation?
No, and this is its most serious weakness. Change the parameterisation, for example estimate a standard deviation instead of a variance, and the MAP estimate changes. The MLE does not have this problem because a likelihood transforms with a Jacobian that cancels in the argmax.
How do I choose the strength of the prior?
Three routes. Cross-validation picks the penalty that predicts best. Empirical Bayes maximises the marginal likelihood over the prior hyperparameters. A weakly informative default, such as a Cauchy prior on standardised coefficients, works when you have no real belief and only want to stop the estimate running to infinity.
Sources
[1]Gelman, Jakulin, Pittau, and Su, A Weakly Informative Default Prior Distribution for Logistic and Other Regression Models (2008) — where the Cauchy(0, 2.5) default and its standardisation come from: stat.columbia.edu/~gelman/research/published/priors11.pdf[2]Park and Casella, The Bayesian Lasso (2008) — the Laplace prior, the scale-mixture construction, and why the posterior mean is not sparse: doi.org/10.1198/016214508000000337[3]Carvalho, Polson, and Scott, The Horseshoe Estimator for Sparse Signals (2010) — global-local shrinkage and the Half-Cauchy hierarchy: arxiv.org/abs/1005.4426[4]scikit-learn, Linear Models — Ridge, Lasso, and ElasticNet, their penalties, and thealphaparameter: scikit-learn.org/stable/modules/linear_model.html[5]Firth, Bias Reduction of Maximum Likelihood Estimates (1993) — the Jeffreys-prior penalty that removes first-order bias and tames separation: doi.org/10.1093/biomet/80.1.27
