Random Forest Algorithm Explained with Python Code

Contents

Figure 1: Hundreds of noisy trees, one calm vote

Bagging, decorrelation, OOB scoring, and the two knobs worth tuning

A single decision tree is easy to understand and easy to fool: change a few training rows and you can get a completely different tree. Random forests turn that weakness into a strength.

Instead of trusting one jumpy tree, they train hundreds of them on slightly different versions of the data and let them vote. The result is one of the most dependable general-purpose models for tabular data, often competitive out of the box with almost no tuning. This guide explains why averaging trees works, builds a miniature forest from scratch, then covers the real scikit-learn workflow, tuning, importance scores, and the limits you should know about.

The Core Idea: Wisdom of Many Noisy Trees

A deep, unpruned decision tree has low bias (it can fit almost anything) but high variance (it's very sensitive to the exact training sample). Averaging many high-variance models reduces variance, as long as the models make different mistakes. A random forest creates that diversity in two ways:

• Bagging (bootstrap aggregating). Each tree trains on a bootstrap sample, a random draw of rows with replacement, the same size as the original data. About 63% of unique rows land in any given sample.
• Random feature subsets. At every split, each tree considers only a random subset of the features (by default, the square root of the feature count for classification). This stops one dominant feature from appearing at the top of every tree, which would make the trees too similar.

For classification, the forest averages the trees' predicted class probabilities (or takes a majority vote). For regression, it averages their predictions.

Why the Randomness Matters

If you average B trees, each with variance σ² and average pairwise correlation ρ, the variance of the average is:

ρσ² + ((1 − ρ) / B) · σ²

Adding trees shrinks the second term toward zero, but the first term, ρσ², is a floor set by how correlated the trees are. That's the whole reason for feature subsampling: it lowers ρ, which lowers the floor. It also explains why adding trees helps with diminishing returns and rarely causes overfitting by itself.

A Forest From Scratch

Here's a minimal classifier built from scikit-learn decision trees, including the out-of-bag (OOB) estimate, a free validation score. Because each tree skips about 37% of rows, you can test every row only on the trees that never saw it:

import numpy as np
from sklearn.tree import DecisionTreeClassifier

class MiniForest:
    """Teaching version. Assumes integer labels 0..k-1 and that every bootstrap
    sample contains every class (true for reasonably sized, balanced data)."""

    def __init__(self, n_trees=100, max_features="sqrt", random_state=0):
        self.n_trees, self.max_features = n_trees, max_features
        self.rng = np.random.default_rng(random_state)

    def fit(self, X, y):
        n = len(X)
        self.classes_ = np.unique(y)
        self.trees, self.oob_masks = [], []
        for _ in range(self.n_trees):
            idx = self.rng.integers(0, n, n)                  # bootstrap sample
            oob = np.ones(n, dtype=bool)
            oob[idx] = False                                  # rows this tree never saw
            tree = DecisionTreeClassifier(
                max_features=self.max_features,               # random subset per split
                random_state=int(self.rng.integers(1_000_000)),
            ).fit(X[idx], y[idx])
            self.trees.append(tree)
            self.oob_masks.append(oob)
        self.X_, self.y_ = X, y
        return self

    def predict_proba(self, X):
        return np.mean([t.predict_proba(X) for t in self.trees], axis=0)

    def predict(self, X):
        return self.classes_[self.predict_proba(X).argmax(axis=1)]

    def oob_score(self):
        votes = np.zeros((len(self.X_), len(self.classes_)))
        counts = np.zeros(len(self.X_))
        for tree, oob in zip(self.trees, self.oob_masks):
            if oob.any():
                votes[oob] += tree.predict_proba(self.X_[oob])
                counts[oob] += 1
        seen = counts > 0
        return (self.classes_[votes[seen].argmax(axis=1)] == self.y_[seen]).mean()

Now compare it with a single tree and scikit-learn's implementation on a dataset bundled with the library:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=0
)

single = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
mini = MiniForest(n_trees=200).fit(X_train, y_train)
sk = RandomForestClassifier(n_estimators=200, oob_score=True,
                            n_jobs=-1, random_state=0).fit(X_train, y_train)

print("single tree     :", round(single.score(X_test, y_test), 3))
print("MiniForest      :", round((mini.predict(X_test) == y_test).mean(), 3),
      "| OOB:", round(mini.oob_score(), 3))
print("sklearn forest  :", round(sk.score(X_test, y_test), 3),
      "| OOB:", round(sk.oob_score_, 3))

On most splits you'll see the forests beat the single tree, and the two forests land close together. The OOB score should track the test score, which is the point: you get an honest generalization estimate without setting aside a validation set.

The Real Thing: A Full scikit-learn Workflow

from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_val_score

rf = RandomForestClassifier(
    n_estimators=300,
    oob_score=True,
    n_jobs=-1,
    random_state=0,
)
rf.fit(X_train, y_train)

proba = rf.predict_proba(X_test)[:, 1]
print(classification_report(y_test, rf.predict(X_test)))
print("ROC AUC:", round(roc_auc_score(y_test, proba), 3))
print("OOB accuracy:", round(rf.oob_score_, 3))

cv = StratifiedKFold(5, shuffle=True, random_state=0)
print("CV AUC:", cross_val_score(rf, X_train, y_train, cv=cv, scoring="roc_auc").mean().round(3))

No scaling step is needed, since trees only care about the order of feature values. That's one reason forests are so low-maintenance.

The Hyperparameters That Matter

• n_estimators: more trees reduce variance with diminishing returns. A few hundred is a common starting point, and more rarely hurts, it just costs time, memory, and prediction latency.
• max_features: the number of features considered per split. Lower values give more diverse (less correlated) trees but weaker individual trees. The default is the square root of the feature count for classification, and it's usually the most influential setting to tune.
• min_samples_leaf and max_depth: forests tolerate deep trees, but larger leaves smooth the predictions, shrink the model, and often help on noisy data. Try values like 1, 3, 5, and 10.
• max_samples: the size of each bootstrap sample as a fraction of the data. Smaller values speed training and add diversity.
• class_weight: use "balanced" or "balanced_subsample" for imbalanced classes, then re-check your decision threshold, as covered in the logistic regression and rare event guides in this series.
• bootstrap and oob_score: bootstrapping is on by default, and OOB scoring requires it.
• n_jobs=-1 parallelizes training across cores, and random_state makes runs reproducible.

A randomized search is an efficient way to tune:

from scipy.stats import randint, uniform
from sklearn.model_selection import RandomizedSearchCV

search = RandomizedSearchCV(
    RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=0),
    param_distributions={
        "max_features": uniform(0.1, 0.6),          # fraction of features per split
        "min_samples_leaf": randint(1, 15),
        "max_depth": [None, 8, 12, 20],
        "max_samples": uniform(0.5, 0.5),           # 50%-100% of rows per tree
    },
    n_iter=25, cv=cv, scoring="roc_auc", random_state=0, n_jobs=-1,
)
search.fit(X_train, y_train)
print(search.best_params_, round(search.best_score_, 3))

Forests are forgiving: the default settings are often within a whisker of the tuned ones, so don't burn hours on a search before you've checked the baseline.

Feature Importance, With Caveats

Scikit-learn exposes feature_importances_, which is the mean decrease in impurity (MDI): how much each feature reduced impurity across all splits, averaged over trees. It's free and fast, but it has known weaknesses. It's computed on training data, so it can credit features that only helped memorize noise. It favors continuous and high-cardinality features. And correlated features split the credit between them.

Permutation importance on held-out data is a more trustworthy check, since it measures how much test performance drops when one feature is shuffled:

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.inspection import permutation_importance

names = load_breast_cancer().feature_names
result = permutation_importance(rf, X_test, y_test, n_repeats=20,
                                random_state=0, scoring="roc_auc", n_jobs=-1)

report = pd.DataFrame({
    "MDI": rf.feature_importances_,
    "permutation": result.importances_mean,
}, index=names).sort_values("permutation", ascending=False)
print(report.head(8))

If the two rankings disagree wildly, trust the permutation version more, and remember that with correlated features, shuffling one can understate its importance because the others cover for it. For per-prediction explanations, SHAP's tree explainers work well on forests, as discussed in the explainability guides earlier in this series.

Regression Forests and Extrapolation

RandomForestRegressor works the same way, with trees that predict a mean and a forest that averages them. It has one important limitation: forests can't extrapolate. A tree's prediction is an average of training targets in a leaf, so the forest can never predict outside the range it saw. If you're forecasting a trending series or predicting values beyond your training range, a forest will plateau at the edge. Models with a linear component, or detrending the target first, handle this better.

Forest vs Single Tree vs Boosting

• Versus a single tree: the forest is far more accurate and stable but loses the readable flowchart. You can inspect importances, but not a simple rule set.
• Versus gradient boosting (XGBoost, LightGBM, CatBoost): boosting builds trees sequentially, each correcting the last, and with good tuning it often edges out forests on accuracy. Forests are more forgiving, train in parallel, and are harder to badly overfit. If you need a strong model quickly with minimal tuning, start with a forest, and if you need the last few points of accuracy, tune a boosted model.
• Versus Extra Trees (ExtraTreesClassifier): this variant picks split thresholds at random as well, which adds more randomness, trains faster, and sometimes generalizes better. It's worth a quick comparison.

Practical Notes

• Probabilities. Averaging tree votes tends to produce probabilities that are less confident than the truth near 0 and 1, so if you need calibrated probabilities for decisions, check a calibration curve and consider CalibratedClassifierCV.
• Missing values and constraints. Recent scikit-learn releases added native handling of missing values in random forests and support for monotonic constraints, but support depends on your installed version, so check the documentation before relying on either.
• Categorical features. Encode them (one-hot or ordinal) before fitting, since scikit-learn forests don't handle raw strings. High-cardinality one-hot features can dilute splits.
• Size and latency. A deep forest of hundreds of trees can be large on disk and slow to predict. Limit depth, raise min_samples_leaf, reduce n_estimators, or compress the model if serving constraints matter.
• Imbalanced data. Use class_weight="balanced_subsample", evaluate with precision, recall, and PR-AUC, and tune the decision threshold with cross-validated predictions.
• Reproducibility. Set random_state, and remember that results can differ slightly across library versions and thread counts.

Common Pitfalls

• Scaling features, which trees don't need.
• Judging the forest on training accuracy, which is usually near 100% and meaningless. Use OOB or held-out scores.
• Over-trusting feature_importances_ from the default MDI.
• Expecting extrapolation from a regression forest.
• Tuning dozens of hyperparameters before checking the defaults.
• Ignoring probability calibration when using predicted probabilities for decisions.
• Leaving class imbalance unaddressed.
• Letting forests grow huge without checking memory and latency.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Why does a random forest use random feature subsets at every split?

Because the variance of an averaged forest is bounded below by ρσ², where ρ is the average correlation between trees. If every tree could pick the same dominant feature first, the trees would be nearly identical, ρ would be high, and adding more trees would stop helping. Restricting each split to a random subset (by default the square root of the feature count for classification) decorrelates the trees, lowers the floor, and makes averaging actually reduce variance.

What is the out-of-bag (OOB) score?

Each bootstrap sample leaves out about 37% of the rows for any given tree, so every training row is tested only on the trees that never saw it. Averaging those predictions gives an honest generalization estimate for free — no separate validation set required. In practice the OOB score should track the held-out test score closely; if it doesn't, something is wrong with your split or your data.

Can a random forest extrapolate?

No. A tree's prediction is the average of the training targets in a leaf, so a forest can never predict outside the range of targets it saw — a trending series will plateau at the edge of the training data. If you need extrapolation, use a model with a linear component or detrend the target first, then model the remainder.

How should I tune a random forest?

Check the defaults first — forests are forgiving and the baseline is often within a whisker of a tuned model. Then run a randomized search over the knobs that matter most: max_features (usually the most influential), min_samples_leaf (larger leaves smooth noisy predictions), max_depth, and max_samples. Use cross-validation with the metric you actually care about, and keep the test set for a single final check.

Wrapping This Up

A random forest trains many deep decision trees on bootstrap samples with random feature subsets, then averages their predictions. Bagging reduces variance, feature subsampling decorrelates the trees, and the out-of-bag rows give you a free validation estimate. In scikit-learn it takes a few lines to train, needs no scaling, and works well with default settings, with max_features and min_samples_leaf as the first knobs to tune and permutation importance as the safer way to read feature relevance.

Is it always the best model? No, boosted trees often win with careful tuning, forests can't extrapolate, and their probabilities need checking. But as a fast, robust, low-maintenance baseline for tabular data, it's hard to beat. Run the code above, compare the single tree to the forest, and watch how much stability the voting buys.

Related Articles

• Decision Trees in Python: Build, Visualize, and Prune
• Machine Learning Algorithms Explained: Complete Beginner's Guide
• Logistic Regression Explained: Build a Classifier Step by Step
• SHAP Values in Python: Explain Your Model's Predictions