Logistic Regression from Scratch with Python Classes

Contents

Figure 1: Logistic Regression from Scratch with Python Classes — a linear score, a sigmoid, a loss, and a class that owns all three

You have typed LogisticRegression().fit(X, y) a hundred times. It works, so why bother opening the hood? Because once you build the class yourself, every scikit-learn argument suddenly makes sense.

This guide shows you how to build logistic regression from scratch with Python classes, using only NumPy. We will design a clean object-oriented API with fit, predict_proba, predict, and score methods. I ran all the code and checked it against scikit-learn, so you can trust the numbers.

How Logistic Regression Works

Logistic regression predicts the probability of a yes-or-no outcome. It takes a weighted sum of your features, then squeezes that sum into a number between 0 and 1. A probability above 0.5 becomes class 1, and anything else becomes class 0.

Three ideas drive the whole algorithm:

1. The linear score: z = X @ w + b combines features with learned weights and a bias.
2. The sigmoid function: 1 / (1 + e^-z) turns any score into a probability.
3. Log loss: this loss function punishes confident wrong answers very hard.

Training means finding the weights that make the log loss as small as possible. We will use gradient descent, which nudges the weights downhill step by step. Why does this work? The log loss forms a smooth bowl, so every step heads toward the bottom.

Why Use a Class?

Functions can solve this problem, but a class keeps everything tidy. The object stores its learned weights, so you can train once and predict many times. It also mirrors the scikit-learn API, which makes your code feel familiar.

Here is the design we will follow:

Method or attribute	Job
__init__	Store hyperparameters like the learning rate and iteration count
fit	Learn the weights and bias from training data
predict_proba	Return probabilities
predict	Turn probabilities into class labels
score	Compute accuracy
weights_, bias_	Learned state, written during fit

Trailing underscores, like weights_, mark values the model learns during fit. Scikit-learn uses the same convention, and it signals what you set and what the model learned.

Step 1: Write the Class Skeleton

Start with the constructor. It stores your settings and creates empty slots for the learned values. We also keep a loss_history_ list so we can watch training progress.

import numpy as np

class LogisticRegression:
    def __init__(self, lr=0.1, n_iters=1000, l2=0.0, tol=1e-8):
        self.lr = lr
        self.n_iters = n_iters
        self.l2 = l2
        self.tol = tol
        self.weights_ = None
        self.bias_ = 0.0
        self.loss_history_ = []

The lr argument controls the step size, and n_iters caps the number of steps. The l2 argument adds optional regularization, and tol lets training stop early once the loss stops improving.

Step 2: Add a Stable Sigmoid

The textbook sigmoid breaks on large negative numbers, because np.exp overflows. A stable version avoids that problem. It computes the formula differently for positive and negative inputs.

    @staticmethod
    def _sigmoid(z):
        return np.where(
            z >= 0,
            1.0 / (1.0 + np.exp(-np.abs(z))),
            np.exp(-np.abs(z)) / (1.0 + np.exp(-np.abs(z))),
        )

The @staticmethod decorator tells Python that this method needs no access to self. Both branches use -np.abs(z), so the exponent never grows large and positive. Scikit-learn and every serious library do something similar.

Step 3: Compute the Loss

Log loss measures how far your predicted probabilities sit from the true labels. We clip the probabilities slightly, because log(0) would blow up the calculation. We also add the optional L2 penalty to the loss.

    def _loss(self, y, p):
        eps = 1e-12
        p = np.clip(p, eps, 1 - eps)
        data_loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
        reg = 0.5 * self.l2 * np.sum(self.weights_ ** 2)
        return data_loss + reg

The leading underscore marks this method as internal. Users will never call it directly. We call it inside fit to record progress.

Step 4: Train With Gradient Descent

Now the heart of the class. Each iteration computes predictions, measures the error, and updates the weights. The gradient has a surprisingly neat form: the feature matrix transposed, times the prediction error, divided by the sample count.

    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights_ = np.zeros(n_features)
        self.bias_ = 0.0
        self.loss_history_ = []

        for _ in range(self.n_iters):
            p = self._sigmoid(X @ self.weights_ + self.bias_)
            error = p - y

            grad_w = (X.T @ error) / n_samples + self.l2 * self.weights_
            grad_b = error.mean()

            self.weights_ -= self.lr * grad_w
            self.bias_ -= self.lr * grad_b

            self.loss_history_.append(self._loss(y, p))
            if (len(self.loss_history_) > 1 and
                    abs(self.loss_history_[-2] - self.loss_history_[-1]) < self.tol):
                break
        return self

Notice the return self at the end. It lets you chain calls like model.fit(X, y).score(X, y). We also leave the bias out of the L2 penalty, which matches common practice.

Step 5: Add Prediction Methods

Prediction reuses the same linear score and sigmoid. predict_proba returns raw probabilities, and predict applies a threshold. A separate threshold argument lets you trade precision for recall later.

    def predict_proba(self, X):
        return self._sigmoid(X @ self.weights_ + self.bias_)

    def predict(self, X, threshold=0.5):
        return (self.predict_proba(X) >= threshold).astype(int)

    def score(self, X, y):
        return np.mean(self.predict(X) == y)

That completes the class. You wrote roughly 60 lines, and you now own a working classifier.

Step 6: Scale Your Features With a Second Class

Gradient descent hates unscaled features. Columns with huge values dominate the gradient and wreck training. A small StandardScaler class fixes that, and it shows off the same fit-then-transform pattern.

class StandardScaler:
    def fit(self, X):
        self.mean_ = X.mean(axis=0)
        self.std_ = X.std(axis=0)
        self.std_[self.std_ == 0] = 1.0
        return self

    def transform(self, X):
        return (X - self.mean_) / self.std_

    def fit_transform(self, X):
        return self.fit(X).transform(X)

The scaler learns the mean and standard deviation from the training data only. You then apply those same values to the test data. That order prevents data leakage.

Step 7: Test It on Real Data

Let us train on scikit-learn's breast cancer dataset. We scale the features first, then fit our class with a learning rate of 0.1 and 1,000 iterations.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

scaler = StandardScaler()
Xtr = scaler.fit_transform(X_tr)
Xte = scaler.transform(X_te)

model = LogisticRegression(lr=0.1, n_iters=1000).fit(Xtr, y_tr)
print(model.score(Xte, y_te))

On my run, the model scored 0.974 accuracy on the test set. The loss started at 0.693, which equals the log of 2 and reflects a coin-flip guess. After 1,000 iterations, it dropped to about 0.057.

That starting value is a neat sanity check. With zero weights, every prediction equals 0.5, and the loss must equal about 0.693. If your first loss looks different, you have a bug.

Check Your Work Against scikit-learn

A from-scratch model deserves a reality check. Scikit-learn minimises a slightly different formula, so you need to match the regularization to compare fairly. Our loss averages over samples, while scikit-learn's C multiplies the summed loss. The conversion is C = 1 / (n_samples * l2).

from sklearn.linear_model import LogisticRegression as SKLogReg

l2 = 0.01
mine = LogisticRegression(lr=0.1, n_iters=5000, l2=l2).fit(Xtr, y_tr)
theirs = SKLogReg(C=1 / (len(y_tr) * l2), max_iter=5000).fit(Xtr, y_tr)

print(mine.score(Xte, y_te), theirs.score(Xte, y_te))

Both models scored 0.982 on my run. Their predicted probabilities differed by at most 0.0024, and the class labels agreed on every test sample. Our 60 lines of NumPy match the library.

I learned something useful along the way. When I first compared an unregularized version against scikit-learn with a huge C, the probabilities disagreed wildly. This dataset separates almost perfectly, so unregularized weights keep growing. Matching the regularization fixed the comparison.

What Happens When Things Go Wrong

Breaking the model teaches you more than running it. Try these experiments.

Skip the Scaling

I trained the same model on the raw features. It scored only 0.632 on the test set, and the final loss sat at about 17.3. Huge feature values made the gradient steps overshoot, so training bounced around instead of settling.

Change the Learning Rate

I trained for 200 iterations at three learning rates and recorded the final loss:

Learning rate	Final loss	What it means
0.001	about 0.454	learning crawls
0.1	about 0.080	works well
1.0	about 0.051	looks fast here

A rate of 1.0 won on this dataset, but I would not trust it everywhere. Larger steps can overshoot and diverge on other data. Start near 0.1 and watch the loss curve.

Add Regularization

Setting l2=0.1 shrank the weight vector's norm from about 3.15 to about 1.11 on my run. Smaller weights usually mean a smoother, less overfit model. The test accuracy landed at 0.956, so you trade a little accuracy for stability on this particular split.

Configuration	Test accuracy	Note
Scaled, lr 0.1, 1,000 iterations	0.974	the baseline run
Scaled, l2 = 0.01, 5,000 iterations	0.982	matches scikit-learn exactly
Unscaled	0.632	loss stuck near 17.3
Scaled, l2 = 0.1	0.956	smaller weights, more stability

Ideas to Extend Your Class

Once the basics work, add more features:

1. Mini-batch training: update weights using small random chunks of data.
2. Multiclass support: train one model per class, or switch to softmax.
3. Class weights: penalize mistakes on a rare class more heavily.
4. Early stopping: stop when validation loss rises.
5. Feature scaling inside fit: bundle the scaler into a pipeline-style class.

Common Pitfalls

I have tripped over each of these:

1. Forgetting to scale features. Training stalls or explodes.
2. Using labels other than 0 and 1. The loss formula assumes 0 and 1.
3. Forgetting the bias term. The model loses its intercept and underfits.
4. Skipping the loss history. Without it, you cannot tell whether training works.
5. Comparing against scikit-learn with mismatched regularization. The results will look broken when they are not.
6. Updating the weights before recording the loss. Log p from the current step, as above, or the curve shifts by one iteration and early stopping fires late.
7. Clipping probabilities only in the loss but not in predict_proba. The two then report slightly different numbers, which is confusing when you debug a single row.

Wrapping Up

Logistic regression boils down to a linear score, a sigmoid, a log loss, and gradient descent. Wrapping those pieces in a class gives you a clean, reusable model that mirrors the tools you already use. And you just proved it matches scikit-learn.

So copy the class, break it on purpose, and watch the loss curve react. Try a new dataset, add mini-batches, or bolt on multiclass support. The best way to understand a model is to build one and then try to wreck it.

If you like this pattern, the random forest built the same way in NumPy extends it to ensembles, and the scikit-learn classification recipes show what the library version looks like once the internals are familiar.

Frequently Asked Questions

Why build logistic regression from scratch instead of using scikit-learn?

Because the library hides four decisions you will eventually have to make yourself: how the sigmoid stays numerically stable, how the loss is clipped, how regularization is scaled, and when training should stop. Building the class takes about 60 lines and turns every scikit-learn argument into something familiar. Keep using the library for real work; build it once for the understanding.

Why does the sigmoid need a stable implementation?

The textbook form 1 / (1 + exp(-z)) overflows for large negative z, because exp of a big positive number exceeds the float range and returns inf. The stable version uses -abs(z) in both branches, so the exponent is never large and positive, and the two branches pick whichever form keeps the arithmetic in range. scikit-learn and SciPy's expit do the same thing internally.

What does the trailing underscore in weights_ mean?

It marks an attribute the model learned during fit, as opposed to a hyperparameter you set in the constructor. Scikit-learn uses the same convention: n_estimators is something you choose, coef_ is something the model produced. Following it makes your class predictable to anyone who has used scikit-learn, and it makes the difference between configuration and learned state obvious at a glance.

Why does unscaled data break gradient descent?

Because the gradient is a sum over features, so a column measured in hundreds of thousands produces updates thousands of times larger than a column measured in fractions. A single learning rate cannot serve both: steps that are sane for the small column explode on the large one. Standardising each column to zero mean and unit variance puts every feature on the same scale, and the same learning rate then works everywhere.

How do I compare my implementation with scikit-learn fairly?

Match the regularization exactly. This implementation averages the loss over samples and adds 0.5 * l2 * ||w||², while scikit-learn multiplies the summed loss by C, so the conversion is C = 1 / (n_samples * l2). Compare with mismatched regularization and the probabilities will diverge wildly even when both implementations are correct, especially on a dataset that separates almost perfectly.

Why did my unregularized model's weights keep growing?

Because the breast cancer data is nearly linearly separable. With no regularization, the loss keeps falling as the weights grow without bound, so training never settles on a finite optimum. Scikit-learn handles the same situation by capping solver iterations and warning about convergence. Adding a small L2 penalty gives the loss a unique minimum and the weights a finite size.

Sources

• [1] scikit-learn, LogisticRegression API reference — C, penalty, max_iter and the solver options that this class imitates in miniature: scikit-learn.org/.../LogisticRegression
• [2] scikit-learn, "Linear Models: logistic regression" — the regularized objective, the role of C, and the binary log-loss formulation used for the comparison: scikit-learn.org/modules/linear_model
• [3] scikit-learn, StandardScaler API reference — why scaling is fitted on training data and only applied to test data, the leakage rule this post follows: scikit-learn.org/.../StandardScaler
• [4] SciPy, scipy.special.expit — the numerically stable sigmoid implementation that motivates the two-branch version here: docs.scipy.org/.../expit
• [5] scikit-learn, breast cancer dataset — the 30 features and binary target behind every accuracy and loss figure in this post: scikit-learn.org/.../load_breast_cancer

Related Articles

• Random Forest from Scratch in Python (NumPy Only)
• Scikit-learn Classification Recipes: Raw Data to Evaluation
• Random Forest Regression Metrics: R², MAE, and RMSE Explained