Interpreting Logistic Regression Coefficients with Regularization and Correlated Features

Contents

Figure 1: Interpreting Logistic Regression Coefficients — one coefficient can look decisive and still be an artefact of its twin

Your logistic regression says feature A raises the odds of churn by 14 times, and feature B cuts them by 81%. Your boss loves the story. Then you retrain on slightly different data, and the numbers flip signs.

That nightmare usually comes from correlated features, and regularization changes the story again. This guide shows you how to read logistic regression coefficients honestly. I ran every experiment below, so the numbers are real.

The Basics: What a Coefficient Means

A logistic regression coefficient measures a change in log-odds. When a feature goes up by one unit and everything else stays fixed, the log-odds of the positive class change by the coefficient. That sentence hides the whole trap, so read it twice.

Log-odds feel unnatural, so most people convert them. Raise e to the power of the coefficient, and you get an odds ratio:

1. Odds ratio above 1: the feature raises the odds of the positive class.
2. Odds ratio below 1: the feature lowers the odds.
3. Odds ratio near 1: the feature barely matters.

An odds ratio of 2 means the odds double, not the probability. Odds and probabilities differ, especially when probabilities sit far from 50%. Keep that distinction in mind whenever you explain a result.

Set Up an Experiment

I built a dataset where I know the truth. Feature x1 drives the outcome, and x3 drives it too. Feature x2 is a near-copy of x1, with a correlation of 0.999, and it has zero real effect. Feature x4 is pure noise.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
n = 1000

x1 = rng.normal(size=n)
x2 = x1 + rng.normal(scale=0.05, size=n)   # near copy of x1
x3 = rng.normal(size=n)                    # independent signal
x4 = rng.normal(size=n)                    # pure noise

logit = 1.0 * x1 + 0.0 * x2 + 0.8 * x3
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

X = np.column_stack([x1, x2, x3, x4])

Now we know the right answers: x1 should get about 1.0, x2 about 0, x3 about 0.8, and x4 about 0. Let us see whether the model recovers them.

Step 1: Standardize Before You Compare

Raw coefficients depend on units. A feature measured in dollars gets a tiny coefficient, while the same feature in millions of dollars gets a huge one. You cannot compare them directly.

To prove the point, I multiplied x3 by 100 and refit. Its coefficient shrank from about 0.67 to about 0.007, with identical predictions. The feature did not become less important. The unit just changed.

Standardize your features with StandardScaler before you compare coefficients. Each coefficient then describes the effect of a one-standard-deviation increase, which puts every feature on the same scale. You should also standardize before using any regularization, because the penalty treats all coefficients equally.

Step 2: Watch Correlated Features Break the Model

Here is the unregularized fit. I set C=1e6, which makes the penalty nearly vanish.

def fit(C, Xm=X, yv=y):
    model = make_pipeline(
        StandardScaler(),
        LogisticRegression(C=C, max_iter=2000),
    )
    return model.fit(Xm, yv)

model = fit(1e6)
coef = model[-1].coef_[0]
print(np.round(coef, 2))
print(np.round(np.exp(coef), 2))

The coefficients came out as 2.65, −1.66, 0.66, and 0.11 for x1 through x4. The odds ratios read 14.1, 0.19, 1.93, and 1.12. Remember the truth: x1 should sit near 1.0, and x2 should sit near zero.

The model gave x1 a huge positive effect and x2 a large negative one. Taken literally, you would conclude that x2 protects against the outcome by 81%. That claim is nonsense, and it comes from correlation alone.

Feature	Truth	Unregularized coefficient	Odds ratio
x1	1.0	2.65	14.1
x2	0.0	−1.66	0.19
x3	0.8	0.66	1.93
x4	0.0	0.11	1.12

Why This Happens

When two features move together, the model cannot tell which one deserves the credit. It only knows that their combined effect matters. Many coefficient pairs fit the data equally well, so the optimizer picks an arbitrary one.

A simple diagnostic confirms the problem. The variance inflation factor (VIF) for x1 and x2 came out around 370. Values above 5 to 10 usually signal trouble, so this one screams.

The Bootstrap Proof

I refit the model on 200 bootstrap resamples to measure stability. The unregularized coefficients for x1 and x2 each wobbled with a standard deviation of about 1.4. The x2 coefficient turned negative in 87.5% of resamples.

Here is the interesting part. The sum of the x1 and x2 coefficients stayed steady, with a mean near 0.99 and a standard deviation of only 0.09. The model knew the combined effect well. It just could not split that effect between the two copies.

Step 3: Add L2 Regularization

Regularization adds a penalty for large coefficients. L2 regularization, the scikit-learn default, shrinks every coefficient toward zero. The C parameter controls the strength, and smaller C means a stronger penalty.

for C in [1e6, 1.0, 0.1, 0.01]:
    coef = fit(C)[-1].coef_[0]
    print(C, np.round(coef, 2))

Here is what happened to the coefficients on x1 through x4:

C	x1	x2	x3	x4
1,000,000	2.65	−1.66	0.66	0.11
1	0.95	0.03	0.65	0.11
0.1	0.53	0.41	0.61	0.11
0.01	0.36	0.35	0.39	0.07

With C=1, the fit looks beautiful: x1 lands near 0.95, x2 near zero, and x3 near 0.65. L2 pulled the wild coefficients back to sensible values. As C shrinks further, x1 and x2 converge to nearly equal values, because L2 splits the credit evenly between correlated twins.

Stability Improves Too

The bootstrap tells the same story. At C=1, the standard deviation for x1 and x2 dropped to about 0.30, and x2 turned negative in 47.5% of resamples. At C=0.05, the standard deviation fell to 0.04, and x2 never turned negative.

Setting	SD of x1/x2 coefficients	x2 negative in bootstrap
Unregularized (C = 1e6)	~1.4	87.5%
C = 1	~0.30	47.5%
C = 0.05	0.04	0%

So regularization buys stability. The price is bias, because the coefficients now underestimate true effects. Notice how x3's coefficient fell from about 0.66 to 0.39 at C=0.01.

Step 4: Try L1 Regularization

L1 regularization behaves differently. It pushes some coefficients exactly to zero, which performs feature selection. Among correlated twins, L1 tends to pick one and drop the other.

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(C=0.1, l1_ratio=1, solver='saga', max_iter=5000),
)
model.fit(X, y)
print(np.round(model[-1].coef_[0], 2))

Scikit-learn 1.8 deprecates the penalty argument in favour of l1_ratio. Older versions use penalty='l1' with solver='liblinear' instead.

On my run, L1 produced these coefficients on x1 through x4:

C	x1	x2	x3	x4
1	0.98	0.00	0.65	0.11
0.1	0.90	0.00	0.58	0.06
0.05	0.82	0.00	0.51	0.01

L1 gave x2 exactly zero at every setting, which matches the truth. It also kept x1 near 0.9 and x3 near 0.6. As the penalty grew, the noise feature x4 faded toward zero.

But beware: L1 picks one twin somewhat arbitrarily. Here it chose the twin that happened to be the real driver, but a different sample could pick the other. That choice does not prove the dropped feature is useless.

Step 5: Merge Correlated Features

Sometimes the cleanest fix skips regularization entirely. If two features carry the same information, combine them. I averaged x1 and x2 into a single feature and refit without any penalty.

X_merged = np.column_stack([(x1 + x2) / 2, x3, x4])
model = fit(1e6, Xm=X_merged)
print(np.round(model[-1].coef_[0], 2))

The coefficients came out at 0.99, 0.65, and 0.11. The merged feature recovered an effect near 1, and nothing flipped sign. You can also drop one twin, or use domain knowledge to build a better combined feature.

How to Interpret Regularized Coefficients

Regularized coefficients need extra care. Keep these rules in mind:

1. Read them as shrunk estimates. The penalty pulls every coefficient toward zero, so odds ratios understate true effects.
2. Compare ranks, not exact sizes. The ordering of strong versus weak features usually survives shrinkage better than the exact values.
3. Never read one twin alone. With correlated features, interpret the group's combined effect.
4. Remember that scikit-learn gives no p-values. The library focuses on prediction, so use a statistics package or the bootstrap when you need confidence intervals.
5. Check stability. Refit on bootstrap samples and look at how much each coefficient moves.

The bootstrap is the most underused tool in this whole process. It takes ten lines of code, and it tells you immediately which coefficients you can trust.

A Practical Workflow

Follow this sequence on real projects:

1. Standardize your features.
2. Check correlations and VIF to spot redundant features.
3. Merge or drop features that carry the same information.
4. Choose C by cross-validation, not by gut feeling.
5. Bootstrap the final model to check coefficient stability.
6. Report odds ratios per standard deviation, with intervals when possible.

This routine takes an extra hour. It also saves you from presenting a coefficient that flips sign next week.

Common Pitfalls

I have seen each of these in real reports:

1. Comparing raw coefficients across different units. Standardize first.
2. Trusting signs when features correlate. A negative coefficient may just compensate for its twin.
3. Reading L1 dropouts as proof of irrelevance. L1 may drop a useful feature when a correlated one stays.
4. Treating regularized odds ratios as exact effects. They carry deliberate bias.
5. Confusing odds with probabilities. An odds ratio of 2 does not double the probability.
6. Choosing C on the test set. It is a hyperparameter like any other; cross-validate it.
7. Reporting a coefficient without the C value, the scaling, and the feature set. Without those three, the number cannot be reproduced or audited.

Wrapping Up

A logistic regression coefficient answers a narrow question: how do the log-odds change when this one feature moves and all others stay fixed? Correlated features break that assumption, and regularization changes the numbers on purpose. Standardize, check correlations, regularize with care, and test stability with the bootstrap.

So take your latest model, compute the VIF for each feature, and run a quick bootstrap on the coefficients. If any sign flips, you just caught a story that would have embarrassed you later. That check beats any fancy plot.

If you want the coefficient extraction mechanics, the from-scratch logistic regression class shows the weights being learned line by line, and the threshold guide covers the next decision once the model is trustworthy. For the broader explainability picture, see the best interpretable machine learning books.

Frequently Asked Questions

What does a logistic regression coefficient actually mean?

It is the change in log-odds for a one-unit increase in that feature, holding every other feature fixed. Exponentiation turns it into an odds ratio: above 1 raises the odds, below 1 lowers them, and near 1 means little effect. The "holding everything else fixed" clause is the fragile part, because correlated features move together and the model cannot isolate one from the other.

Why do coefficients flip sign when features are correlated?

Because many different coefficient pairs fit the data almost equally well. When two columns carry the same information, the optimizer can give one a large positive weight and the other a large negative one, and the combined effect stays correct. Retrain on a slightly different sample and the split can reverse. The sum of the pair is stable while the individual signs are not, which is exactly what a bootstrap reveals.

What VIF value means a collinearity problem?

A common rule of thumb is that VIF above 5 to 10 signals trouble, because it means the feature is largely explained by the others. In this experiment the near-duplicate columns scored around 370, which is severe. VIF is a diagnostic, not a verdict: it tells you the coefficient is unstable, and the fix is to merge, drop, or regularize rather than to stare at the number.

Does L2 regularization fix collinearity?

It stabilises the coefficients without resolving the ambiguity. L2 shrinks both twins toward zero and, as the penalty grows, splits the credit between them almost equally, so the signs stop flipping but the individual values stop being interpretable as separate effects. It buys stability at the price of bias: in this run L2 pulled a coefficient of 0.66 down to 0.39, so odds ratios understate the true effect.

Why did L1 regularization zero out one of two identical features?

L1 penalises the absolute size of coefficients, so the cheapest solution is often to keep one twin and drop the other entirely. Which one survives is somewhat arbitrary and can change with a different sample, so a zero coefficient is not proof that the feature is useless. Treat L1 as a selector, then check whether the surviving feature is the one your domain knowledge expects.

Should I report odds ratios from a regularized model?

Only with a clear caveat, and ideally with intervals from a bootstrap. Regularized coefficients are deliberately biased toward zero, so the odds ratios describe a shrunk model rather than the data-generating process. Report them as relative rankings and directional effects, state the C value used, and never present a single regularized odds ratio as an exact causal effect.

Sources

• [1] scikit-learn, LogisticRegression API reference — C, l1_ratio, the penalty deprecation, and the saga solver that supports both L1 and L2: scikit-learn.org/.../LogisticRegression
• [2] scikit-learn, "Linear Models: logistic regression" — the penalised objective, the meaning of C, and how L1 and L2 differ in what they do to coefficients: scikit-learn.org/modules/linear_model
• [3] statsmodels, variance_inflation_factor — the VIF diagnostic used to quantify how much of each feature is explained by the others: statsmodels.org/.../variance_inflation_factor
• [4] scikit-learn, "Feature selection: L1-based selection" — why the L1 penalty produces exact zeros, and the caveat about arbitrarily choosing among correlated features: scikit-learn.org/modules/feature_selection
• [5] James, Witten, Hastie and Tibshirani, "An Introduction to Statistical Learning" — free PDF covering logistic regression, odds and the logit link, plus the ridge and lasso chapters: statlearning.com

Related Articles

• Logistic Regression from Scratch with Python Classes
• How to Choose the Classification Threshold in Logistic Regression
• The Best Interpretable Machine Learning Books