Random Forest from Scratch in Python (NumPy Only)

Contents

Figure 1: Random Forest from Scratch in Python (NumPy Only) — a tree, a bootstrap sample, a feature subset, and a vote

You call RandomForestClassifier().fit() and it works. Great. But do you know why it works? I did not, until I built one myself with nothing but NumPy.

This guide shows you how to build a random forest from scratch in Python using only NumPy. We will write a decision tree, add randomness, and combine many trees into a forest. The whole thing fits in about 100 lines, and I tested every line before I wrote this.

How a Random Forest Works

A random forest combines many decision trees into one stronger model. Each tree sees a slightly different slice of the data and votes on the answer. The forest picks the label that wins the most votes.

Two tricks make the trees different from each other:

1. Bootstrap sampling: each tree trains on a random sample of rows, drawn with replacement. Some rows repeat and some disappear.
2. Random feature selection: at each split, the tree considers only a random subset of features.

Why does this help? One deep tree memorizes its training data and overfits. Many different trees make different mistakes, and voting cancels out much of the noise.

Step 1: Measure Impurity With Gini

A decision tree needs a way to judge a split. We use Gini impurity, which measures how mixed the classes are in a group. A pure group scores 0, and a 50/50 mix of two classes scores 0.5.

The formula is simple: one minus the sum of squared class proportions. NumPy handles it in a few lines.

import numpy as np

def gini(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / counts.sum()
    return 1.0 - np.sum(p ** 2)

Step 2: Build the Decision Tree

The tree splits the data again and again until it hits a stopping rule. Each split asks a yes-or-no question like "is feature 3 below 0.7?" The leaves hold the final predictions.

Create a Node Class

A node stores either a split rule or a final answer. Internal nodes hold a feature, a threshold and two children. Leaf nodes hold a predicted class in value.

class Node:
    def __init__(self, feature=None, threshold=None, left=None, right=None, value=None):
        self.feature = feature
        self.threshold = threshold
        self.left = left
        self.right = right
        self.value = value

Find the Best Split

The tree tries many candidate splits and keeps the best one. For each feature, it tests every unique value as a threshold. It then computes the weighted Gini impurity of the two resulting groups.

The information gain equals the parent's impurity minus the children's weighted impurity. The split with the biggest gain wins. Notice the features line: it picks a random subset, which brings the "random" into random forest.

class DecisionTree:
    def __init__(self, max_depth=10, min_samples_split=2, n_features=None, rng=None):
        self.max_depth = max_depth
        self.min_samples_split = min_samples_split
        self.n_features = n_features
        self.rng = rng or np.random.default_rng()
        self.root = None

    def _best_split(self, X, y):
        n_samples, n_total = X.shape
        k = self.n_features or n_total
        features = self.rng.choice(n_total, size=k, replace=False)
        best_gain, best_feature, best_threshold = 0.0, None, None
        parent = gini(y)
        for f in features:
            for t in np.unique(X[:, f])[:-1]:
                mask = X[:, f] <= t
                n_left = mask.sum()
                n_right = n_samples - n_left
                child = (n_left * gini(y[mask]) + n_right * gini(y[~mask])) / n_samples
                gain = parent - child
                if gain > best_gain:
                    best_gain, best_feature, best_threshold = gain, f, t
        return best_feature, best_threshold

Grow the Tree Recursively

The _grow method builds the tree one split at a time. It stops when the depth limit hits, when too few samples remain, or when all labels match. At that point it creates a leaf with the most common class.

Both methods below belong to the same DecisionTree class, so keep them indented at the class level.

    def fit(self, X, y):
        self.root = self._grow(X, y, depth=0)
        return self

    def _grow(self, X, y, depth):
        if (depth >= self.max_depth or len(y) < self.min_samples_split
                or len(np.unique(y)) == 1):
            return Node(value=np.bincount(y).argmax())
        feature, threshold = self._best_split(X, y)
        if feature is None:
            return Node(value=np.bincount(y).argmax())
        mask = X[:, feature] <= threshold
        left = self._grow(X[mask], y[mask], depth + 1)
        right = self._grow(X[~mask], y[~mask], depth + 1)
        return Node(feature, threshold, left, right)

np.bincount needs integer labels starting at 0. Encode your classes as 0, 1, 2 and so on before you train.

Make Predictions

Prediction walks one sample down the tree. At each node, the sample goes left or right based on the threshold. It stops at a leaf and returns that leaf's class.

    def _predict_one(self, x):
        node = self.root
        while node.value is None:
            node = node.left if x[node.feature] <= node.threshold else node.right
        return node.value

    def predict(self, X):
        return np.array([self._predict_one(x) for x in X])

Step 3: Build the Forest

Now the fun part. The forest trains many trees, and each one gets its own bootstrap sample. By default, each split considers the square root of the total feature count, a common choice for classification.

class RandomForest:
    def __init__(self, n_trees=50, max_depth=10, min_samples_split=2,
                 n_features=None, seed=42):
        self.n_trees = n_trees
        self.max_depth = max_depth
        self.min_samples_split = min_samples_split
        self.n_features = n_features
        self.rng = np.random.default_rng(seed)
        self.trees = []

    def fit(self, X, y):
        n_samples, n_total = X.shape
        k = self.n_features or int(np.sqrt(n_total))
        self.trees = []
        for _ in range(self.n_trees):
            idx = self.rng.integers(0, n_samples, size=n_samples)
            tree = DecisionTree(self.max_depth, self.min_samples_split, k, self.rng)
            tree.fit(X[idx], y[idx])
            self.trees.append(tree)
        return self

    def predict(self, X):
        votes = np.array([tree.predict(X) for tree in self.trees])
        return np.array([np.bincount(col).argmax() for col in votes.T])

The idx line performs the bootstrap: it draws row numbers with replacement. The predict method collects every tree's votes and picks the majority for each sample. That is the entire algorithm.

Step 4: Test It on Real Data

Let us check whether the forest earns its keep. We will generate a small synthetic dataset, train a single tree, and compare it with the forest. Same data, same split, fair fight.

def accuracy(y_true, y_pred):
    return np.mean(y_true == y_pred)

rng = np.random.default_rng(0)
n = 600
X = rng.normal(size=(n, 6))
y = ((X[:, 0] + X[:, 1] ** 2 - X[:, 2]
      + rng.normal(scale=0.5, size=n)) > 1).astype(int)

idx = rng.permutation(n)
split = int(0.8 * n)
tr, te = idx[:split], idx[split:]

tree = DecisionTree(max_depth=10, rng=np.random.default_rng(1)).fit(X[tr], y[tr])
forest = RandomForest(n_trees=50, max_depth=10).fit(X[tr], y[tr])

print('tree', accuracy(y[te], tree.predict(X[te])))
print('forest', accuracy(y[te], forest.predict(X[te])))

Model	Accuracy on the held-out 20%
Single decision tree	0.80
Random forest, 50 trees	0.875

On my run, the single tree scored 0.80 and the forest scored 0.875. Your numbers may differ slightly with other data or seeds. The pattern should hold, though: the forest beats the lone tree because voting smooths out individual mistakes.

Why Your Version Runs Slower Than scikit-learn

You will notice that this code crawls on big datasets. Scikit-learn runs its tree-building loops in compiled code and uses smarter tricks. Our version recomputes Gini from scratch for every candidate split.

Here are the main reasons for the gap:

1. Python loops: we loop over thresholds in pure Python, which runs slowly.
2. Repeated work: we rescan the data for every threshold instead of sorting once.
3. No parallelism: we train trees one at a time, while scikit-learn can use multiple cores.

That speed gap is fine. We built this to learn, not to compete with production code. Use scikit-learn for real projects, and use this version to understand what happens inside.

Ideas to Extend Your Forest

Once the basics work, try these upgrades. Each one teaches you something new.

1. Out-of-bag score: test each tree on the rows its bootstrap sample skipped. You get a free validation estimate without a separate test set.
2. Feature importance: add up the information gain each feature produces across all splits.
3. Probability outputs: return the share of votes per class instead of a single label.
4. Faster splits: sort each feature once and compute Gini with running counts.
5. Regression support: swap Gini for variance and average the leaf values.

Common Pitfalls

I hit most of these while building mine. Save yourself the headache.

1. Forgetting the bootstrap. Without it, every tree looks nearly identical and the forest adds little.
2. Using all features at every split. You lose the randomness that makes trees diverse.
3. Using string labels. np.bincount expects non-negative integers. Encode your labels first.
4. Setting no depth limit. Trees can grow huge and slow. Cap max_depth or min_samples_split.
5. Sharing one RNG carelessly. Passing the same generator into every tree is fine, but reusing a seeded one across experiments makes runs look reproducible when they are not reproducible for the same data order.
6. Forgetting [:-1] on the thresholds. Splitting at the maximum value of a feature puts every sample on the left, so the right child is empty and the recursion stalls.
7. Judging the forest on training accuracy. Both models will look excellent there, which tells you nothing about generalisation.

Wrapping Up

A random forest is just a decision tree, a bootstrap sample, a random feature subset, and a vote. You now own all four pieces, and you built them yourself. Next time fit() runs, you will know exactly what it does.

So copy the code, break it, and fix it. Try a different dataset, add out-of-bag scoring, or speed up the splits. That tinkering teaches more than any tutorial, including this one.

If you want the library version next to this one, the scikit-learn classification recipes show the production workflow around the same algorithm: pipelines, cross-validation and honest metrics. The same instinct — build the small thing yourself so the framework stops being magic — is what the low-level MCP server walkthrough does for protocol work.

Frequently Asked Questions

What is the difference between a decision tree and a random forest?

A decision tree is one model that splits the data until a stopping rule and predicts a single leaf label. A random forest trains many trees, each on a bootstrap sample of rows and each considering only a random subset of features at every split, then predicts by majority vote. The trees are individually worse and collectively better, because their mistakes disagree and the vote cancels much of the noise.

Why does a random forest use bootstrap sampling?

Bootstrap sampling draws rows with replacement, so each tree sees roughly 63% of the unique rows and a different mix every time. That variation is what makes the trees disagree, and disagreement is what voting exploits. Without bootstrapping, every tree would be trained on identical data, look nearly identical, and the forest would gain little over a single tree.

Why use only a random subset of features at each split?

Because a dominant feature would otherwise appear in the first split of every tree, making them look alike. Restricting each split to a random subset, commonly the square root of the total feature count for classification, forces different trees to find different explanations. That decorrelation is the second half of what makes a forest better than one tree, alongside the bootstrap sample.

Why is np.bincount used for labels?

It counts occurrences of each non-negative integer, which makes majority voting a one-liner and leaf predictions fast. The catch is that it requires integer labels starting at zero, so encode classes as 0, 1, 2 before training. String labels or negative integers will raise, and sparse class numbers create long zero-filled arrays for no reason.

How do I make my from-scratch random forest faster?

Sort each feature once and compute Gini with running counts instead of rescanning the data for every threshold, which is the dominant cost in pure Python. Then consider vectorising the threshold search across features, and training trees in parallel processes. A compiled implementation like scikit-learn will still be faster, because its splitter runs in Cython and uses presorted indices.

Should I use my own implementation in production?

No. Build it to understand the algorithm, then use scikit-learn for real work: it is faster, tested, and handles edge cases you have not thought about. Your version is valuable because you now know what fit() actually does, and because the out-of-bag score and feature importance ideas in this guide transfer directly to the library version.

Sources

• [1] scikit-learn, "Ensembles: forests of randomized trees" — how bagging and feature subsampling combine, the max_features="sqrt" default for classification, and out-of-bag estimates: scikit-learn.org/modules/ensemble
• [2] scikit-learn, "Decision trees" — Gini impurity, how a splitter evaluates thresholds, and the stopping criteria this implementation mirrors: scikit-learn.org/modules/tree
• [3] Leo Breiman, "Random Forests" (Machine Learning, 2001) — the original paper defining the algorithm, the bootstrap sample and random feature selection at each split: link.springer.com
• [4] Hastie, Tibshirani and Friedman, "The Elements of Statistical Learning" — the tree-based methods chapter covering bagging, variance reduction and why ensembles of randomised trees work: hastie.su.domains/ElemStatLearn
• [5] NumPy, random generator and np.bincount documentation — default_rng, choice without replacement, integers for bootstrap indices, and the non-negative integer requirement for counting: numpy.org/random, numpy.bincount

Related Articles

• Scikit-learn Classification Recipes: Raw Data to Evaluation
• Official MCP Python SDK: Low-Level Server Without FastMCP
• Best LLM APIs for Tool Calling and Function Calling
• Validating Tool Call Outputs with Pydantic