Logistic Regression Explained: Build a Classifier Step by Step

Contents

Figure 1: A probability, a threshold, a decision — the entire classifier in three steps

Sigmoid intuition, a from-scratch NumPy build, and a complete scikit-learn workflow with tuned thresholds

Logistic regression has a misleading name and a loyal following. It's a classifier, not a regression model in the "predict a number" sense, and it's usually the first model serious practitioners reach for on a new classification problem.

It's fast, its predictions come with probabilities, and you can read its reasoning straight off the coefficients. This guide builds one in three passes: the intuition, a from-scratch version in NumPy so nothing feels like magic, and a complete scikit-learn workflow with evaluation, tuning, and threshold selection. Every number comes from a script run with NumPy 2.5.3 and scikit-learn 1.9.1.

What Logistic Regression Actually Does

Imagine predicting whether a tumor is malignant from measurements. A linear model could compute a score:

z = b + w₁x₁ + w₂x₂ + … + wₙxₙ

That score can be any number, but we need a probability between 0 and 1. The sigmoid function squashes any real number into a probability between 0 and 1:

p = 1 / (1 + e^(−z))

Large positive scores give probabilities near 1, large negative scores near 0, and a score of zero gives exactly 0.5. Then you apply a threshold, commonly 0.5, to turn the probability into a class.

Here's the idea that makes the model interpretable. Rearranging the formula shows that the score z is the log-odds — the natural log of p / (1 − p). Each coefficient says how much the log-odds change per unit increase in that feature. Exponentiate a coefficient and you get an odds ratio: if e^w equals 1.5, a one-unit increase in that feature multiplies the odds of the positive class by 1.5, holding the others fixed.

How It Learns

Training means finding the weights that make the observed labels most probable. That's done by minimizing log loss (also called cross-entropy), which punishes confident wrong predictions very heavily: predicting 0.99 for something that turns out to be negative costs far more than predicting 0.6.

Unlike linear regression, there's no closed-form answer, so an optimizer such as gradient descent or L-BFGS iterates toward the minimum. The log-loss surface is convex, so you reach the global optimum, not a local trap.

Regularization adds a penalty on large weights to prevent overfitting. L2 shrinks weights smoothly, L1 can drive some to exactly zero (feature selection), and elastic net mixes the two. In scikit-learn the strength is controlled by C, which is the inverse of regularization strength: smaller C means stronger regularization.

Part 1: A From-Scratch Version

Writing it yourself takes about 20 lines, and it demystifies everything that follows:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -30, 30)))     # clip avoids overflow

class SimpleLogReg:
    def __init__(self, lr=0.1, epochs=2000, l2=0.01):
        self.lr, self.epochs, self.l2 = lr, epochs, l2

    def fit(self, X, y):
        n, d = X.shape
        self.w, self.b = np.zeros(d), 0.0
        for _ in range(self.epochs):
            p = sigmoid(X @ self.w + self.b)
            error = p - y                  # gradient of log loss with respect to the score
            self.w -= self.lr * (X.T @ error / n + self.l2 * self.w)
            self.b -= self.lr * error.mean()
        return self

    def predict_proba(self, X):
        return sigmoid(X @ self.w + self.b)

    def predict(self, X, threshold=0.5):
        return (self.predict_proba(X) >= threshold).astype(int)

The whole algorithm is the line error = p - y: the gradient of log loss with respect to the score is simply prediction minus truth, and the weights move against it.

Gradient descent needs standardized features to converge sensibly, which is also true for regularization to treat features fairly — keep that in mind for the scikit-learn version, where the library does it for you.

Part 2: Step-by-Step with Scikit-learn

Step 1: Load the data and choose the positive class

We'll use the breast cancer dataset bundled with scikit-learn. One detail trips people up: in this dataset, label 0 means malignant and label 1 means benign. Decide which class you care about and make that the positive class (1), because precision and recall are defined relative to it. Here we want to catch malignant cases:

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer(as_frame=True)
X = data.data
y = (data.target == 0).astype(int)        # 1 = malignant (the class we want to catch)

print(y.value_counts(normalize=True))
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

Stratifying keeps the class proportions the same in both splits, which matters when classes are imbalanced.

Step 2: Build a pipeline and fit

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
pipe.fit(X_train, y_train)

The pipeline fits the scaler on training data only and applies the same transformation at prediction time, preventing leakage. Raising max_iter avoids convergence warnings on larger problems.

Step 3: Evaluate honestly

from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score

proba = pipe.predict_proba(X_test)[:, 1]
pred = pipe.predict(X_test)

print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=["benign", "malignant"]))
print("ROC AUC:", round(roc_auc_score(y_test, proba), 3))

Read the confusion matrix first: it shows how many malignant cases were missed (false negatives) and how many benign cases were flagged (false positives). In a medical setting a missed malignancy is the costlier mistake, so recall on the malignant class matters most. Accuracy alone can hide this, especially with imbalanced classes, as covered in the rare event detection guide in this series.

Step 4: Read the coefficients

Because we standardized the features, each coefficient describes the effect of a one-standard-deviation increase, which makes features comparable:

clf = pipe.named_steps["logisticregression"]
table = pd.DataFrame({
    "coef": clf.coef_[0],
    "odds_ratio_per_SD": np.exp(clf.coef_[0]),
}, index=X.columns).sort_values("coef")

print(pd.concat([table.head(5), table.tail(5)]))

Positive coefficients push toward malignant, negative ones toward benign. Treat these carefully when features are strongly correlated, since the model may split credit between them arbitrarily, and remember that coefficients describe the model, not causes.

Step 5: Tune regularization with cross-validation

from sklearn.model_selection import GridSearchCV, StratifiedKFold

grid = GridSearchCV(
    make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)),
    param_grid={"logisticregression__C": [0.01, 0.1, 1, 10, 100]},
    scoring="roc_auc",
    cv=StratifiedKFold(5, shuffle=True, random_state=0),
)
grid.fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))

Tuning happens on the training data via cross-validation, and the test set stays untouched for the final check.

Step 6: Choose a threshold for your actual costs

The default 0.5 threshold isn't sacred. If missing a malignant case is much worse than a false alarm, lower the threshold to catch more positives at the cost of precision. Pick the threshold using cross-validated predictions on the training set, not the test set, or you leak information:

from sklearn.metrics import precision_recall_curve
from sklearn.model_selection import cross_val_predict

best = grid.best_estimator_
oof = cross_val_predict(best, X_train, y_train, cv=5, method="predict_proba")[:, 1]

precision, recall, thresholds = precision_recall_curve(y_train, oof)
ok = np.where(recall[:-1] >= 0.98)[0]            # thresholds that keep recall >= 98%
threshold = thresholds[ok[-1]] if len(ok) else 0.5

final_pred = (best.predict_proba(X_test)[:, 1] >= threshold).astype(int)
print("chosen threshold:", round(float(threshold), 3))
print(classification_report(y_test, final_pred, target_names=["benign", "malignant"]))

The target of 98% recall here is a stand-in for whatever your real cost tradeoff dictates. The point is to decide the threshold deliberately, validate it on held-out data, and report precision alongside it.

Step 7: Save the model

import joblib
joblib.dump(best, "malignancy_logreg.joblib")

Because the scaler lives in the pipeline, the saved object handles preprocessing too, so a single file is enough at prediction time.

A Note on scikit-learn's Changing API

Many tutorials pass penalty="l1" or penalty="l2". In scikit-learn 1.8 that parameter was deprecated and is scheduled for removal in version 1.10. The documentation says to use l1_ratio=0 in place of penalty="l2", l1_ratio=1 for "l1", and values between 0 and 1 for elastic net, and the same page indicates C=np.inf stands in for having no penalty.

So an L1 model on a recent version looks like LogisticRegression(l1_ratio=1, solver="saga"), while older versions need the old penalty="l1" argument. Two other changes are worth knowing: n_jobs is deprecated and has no effect, and the old multi_class option was removed in favor of always using the multinomial loss for three or more classes, with OneVsRestClassifier available if you want one-vs-rest. Check the documentation for your installed version, since the default solver, lbfgs, is also what most examples rely on. — scikit-learn documentation

Assumptions and Limits

Logistic regression assumes the log-odds are a linear function of your features. Several other points follow:

• Non-linear relationships need help: add polynomial terms, splines, interactions, or binned features, or switch to trees or boosting, as covered in the algorithms guide in this series.
• Independent observations: repeated measurements on the same subject violate this and inflate confidence.
• Multicollinearity makes individual coefficients unstable, even when predictions are fine. Regularization helps.
• Sample size: a common rule of thumb asks for roughly ten events of the rarer class per predictor, though regularization relaxes this.
• Perfect separation, where one feature perfectly predicts the label, makes unregularized coefficients grow without bound. Regularization fixes this too.
• Calibration: logistic regression usually produces reasonably calibrated probabilities, but class weighting, strong regularization, and distribution shift can distort them. Check a calibration curve before treating outputs as true risks.

Extensions Worth Knowing

Multiclass classification uses the multinomial (softmax) form, which scikit-learn applies automatically for three or more classes in current versions.

Imbalanced data: class_weight="balanced" reweights the loss toward the minority class. It tends to raise recall and lower precision, so reevaluate your threshold afterward, and consider resampling carefully as discussed in the rare event guide.

Regularization choices: L2 is a sensible default, L1 helps when you suspect many irrelevant features, and elastic net blends both. LogisticRegressionCV bundles cross-validated selection of C.

Feature engineering is where logistic regression gets its power. Interaction terms, log transforms, and domain-specific features often matter more than the choice of algorithm.

When to Use It

Logistic regression is the right first model when you want a fast, strong baseline, need explainable decisions in settings like credit or healthcare, have modest data, or need probability estimates.

It's also a useful yardstick: if gradient boosting beats it by only a tiny margin, the simpler model often wins on maintainability and trust. When relationships are strongly non-linear or full of interactions, move to ensembles, or try an Explainable Boosting Machine if you want accuracy and interpretability together.

Common Pitfalls

• Forgetting to scale features, which hurts convergence and makes regularization treat features unequally.
• Fitting the scaler on the full dataset before splitting, leaking test statistics into training.
• Reporting accuracy on imbalanced data when precision and recall tell the real story.
• Defining the positive class carelessly, as the breast cancer labels show — precision and recall are relative to it.
• Reading coefficients from unscaled features as if they were comparable across features.
• Using 0.5 as the threshold without thinking about costs, when false positives and false negatives are not equally bad.
• Tuning the threshold or hyperparameters on the test set, which quietly inflates every number you report.
• Treating odds ratios as causal effects — they describe the model's fit, not the world.
• Copying penalty= arguments from an old tutorial into a recent scikit-learn version, where they may warn or fail.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Why is it called logistic regression if it's a classifier?

Because the model itself performs a regression — on the log-odds. The sigmoid maps the linear score z to a probability, and z is the log-odds of the positive class, so each coefficient is literally a change in log-odds per unit of feature. The "regression" refers to fitting that continuous quantity; the classification only happens when you apply a threshold, commonly 0.5, to the resulting probability.

What does C control in scikit-learn's LogisticRegression?

C is the inverse of regularization strength: smaller C means stronger regularization. The penalty keeps weights from growing to fit noise — L2 shrinks them smoothly, L1 can drive some to exactly zero, and elastic net mixes both. Tune it on a log grid (0.01 to 100) with cross-validation as in the GridSearchCV example, and remember the tuning happens on training data only: the test set stays untouched for the final check.

How do I choose a threshold other than 0.5?

Start from your costs, not from convention. If a false negative — missing a malignant case — is far worse than a false alarm, you want higher recall and a lower threshold. Pick the threshold from a precision-recall curve computed on cross-validated out-of-fold predictions of the training set, never the test set, or you leak information. In this article, thresholds keeping recall at or above 98% on out-of-fold predictions were selected, and the final report was computed once on the test set.

Why do features need to be standardized first?

Two reasons. Gradient-based optimizers converge far more reliably when features share a scale, and regularization penalizes coefficients uniformly — with unscaled features, a large-unit feature is treated differently from a small-unit one, making the penalty unfair and the coefficients incomparable. StandardScaler inside the pipeline fits on training data only (no leakage), and because scaling happens before the fit, each coefficient describes the effect of a one-standard-deviation change, which makes feature comparison meaningful.

Wrapping This Up

Logistic regression turns a weighted sum of features into a probability through the sigmoid function, learns by minimizing log loss, and explains itself through odds ratios. A complete workflow takes about a dozen lines: split with stratification, put scaling and the model in a pipeline, evaluate with a confusion matrix and class-specific metrics, tune C with cross-validation, choose a threshold using training-set predictions, and save the whole pipeline.

Will it beat a boosted ensemble on every dataset? No, but it will often come close, and it will tell you why it decided what it decided. Run the code above, change C, shift the threshold, and watch how precision and recall trade off. That small experiment teaches more about classification than any amount of reading.

Related Articles

• Linear Regression in Python: From Scratch to Scikit-learn
• Machine Learning Algorithms Explained: Complete Beginner's Guide
• Synthetic Data for Rare Event Detection: Fraud, Defects, and Anomalies
• Imbalanced-learn (imblearn): Handle Imbalanced Datasets...