Random Forest Regression Metrics: R², MAE, and RMSE Explained

Contents

Figure 1: Random Forest Regression Metrics — three metrics, three different questions about the same predictions

Your random forest regressor just reported an R² of 0.44. Is that good? Bad? Should you panic or celebrate? Nobody tells you, and the metric names do not help.

This guide explains the three metrics you will meet most often: R², MAE, and RMSE. You will learn what each one measures, how to compute it, and how to pick the right one. I ran every code block below, so the numbers are real.

Why Regression Metrics Matter

A random forest regressor predicts a number, such as a price, a temperature, or a disease progression score. You need to know how far off those predictions land. One metric rarely tells the whole story.

Think of the three metrics as three different questions:

• MAE: On average, how far off is each prediction?
• RMSE: How far off are the predictions when big mistakes count extra?
• R²: How much better is the model than just guessing the average?

Let us set up a model and compute all three. Then we will decode the results.

Set Up a Random Forest Regressor

We will use scikit-learn's diabetes dataset, which ships with the library. The target measures disease progression one year after baseline, and it has a mean of about 152 and a standard deviation of about 77. Keep those two numbers in mind, because they give every error some context.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

X, y = load_diabetes(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, random_state=42
)

rf = RandomForestRegressor(n_estimators=200, random_state=42)
rf.fit(X_tr, y_tr)
pred = rf.predict(X_te)

Now we have predictions on data the model never saw. Time to measure them.

MAE: Mean Absolute Error

MAE averages the size of your errors, ignoring direction. You take each prediction error, drop the minus sign, and average the results. The answer uses the same units as your target.

mae = mean_absolute_error(y_te, pred)
print(round(mae, 2))   # 44.28

Our forest scored an MAE of 44.28. That means predictions miss the true value by about 44 points on average. Compare that with the target's standard deviation of 77, and you can judge whether 44 feels big or small.

Why People Like MAE

MAE is easy to explain. You can tell a non-technical colleague, "We miss by about 44 points on average," and they will understand. It also treats every error equally, so one wild miss does not dominate the score.

RMSE: Root Mean Squared Error

RMSE squares each error, averages the squares, and then takes the square root. Squaring punishes large errors much harder than small ones. Taking the root brings the result back to the target's original units.

rmse = np.sqrt(mean_squared_error(y_te, pred))
print(round(rmse, 2))   # 54.46

Our RMSE came out at 54.46, higher than the MAE of 44.28. That gap tells you something. RMSE always sits at or above MAE, and a bigger gap means a few large errors drag the score up.

MAE vs RMSE: A Tiny Experiment

Here is a quick example that shows the difference. Imagine two models that each make 10 predictions.

Model	Errors	MAE	RMSE
A	5 on every prediction	5.0	5.0
B	1 on nine predictions, 41 on one	5.0	13.0

Both models share the same MAE. Yet Model B hides one terrible miss, and RMSE exposes it. Which model would you trust with a bridge-load estimate? Exactly.

R²: The Coefficient of Determination

R² compares your model against the laziest possible model: one that always predicts the average. A score of 1 means perfect predictions. A score of 0 means you did no better than guessing the mean.

r2 = r2_score(y_te, pred)
print(round(r2, 3))   # 0.44

Our forest scored an R² of 0.44. In plain terms, the model explains about 44% of the variation in the target. The other 56% stays unexplained.

R² can go negative. A model that performs worse than the mean-guesser scores below zero. That result usually signals a bug or a badly overfit model.

Is 0.44 Good?

It depends on the problem. Medical and social data often contain heavy noise, so modest R² values are normal there. Physics experiments with clean measurements demand much higher scores.

Never judge R² against a universal threshold. Judge it against a baseline and against what your field considers reasonable.

Always Compare Against a Baseline

A metric means little without a reference point. Scikit-learn's DummyRegressor predicts the training mean every time, which gives you the lazy benchmark. Run it first.

from sklearn.dummy import DummyRegressor

base = DummyRegressor(strategy='mean').fit(X_tr, y_tr)
base_pred = base.predict(X_te)

Here is how the forest compared with the baseline on my run:

Metric	Random forest	Mean baseline
MAE	44.28	64.01
RMSE	54.46	73.22
R²	0.44	about −0.01

The forest cuts the average error by roughly a third. That comparison makes the numbers meaningful. The baseline's near-zero R² also confirms how R² works: guessing the mean earns you zero.

Compute the Metrics by Hand

Understanding beats memorising. Each metric takes only a line or two of NumPy. I confirmed these match scikit-learn exactly.

resid = y_te - pred

mae  = np.mean(np.abs(resid))
rmse = np.sqrt(np.mean(resid ** 2))
r2   = 1 - np.sum(resid ** 2) / np.sum((y_te - y_te.mean()) ** 2)

The R² formula reveals its meaning. The top part sums your squared errors. The bottom part sums the squared errors of the mean-guesser. R² measures how much smaller the first sum is than the second.

Use Cross-Validation for Honest Scores

A single train-test split can mislead you. Cross-validation averages results across several splits. Scikit-learn uses negative error names so that higher always means better.

from sklearn.model_selection import cross_val_score

model = RandomForestRegressor(n_estimators=200, random_state=42)

for scoring in ['neg_mean_absolute_error',
                'neg_root_mean_squared_error',
                'r2']:
    scores = cross_val_score(model, X, y, cv=5, scoring=scoring)
    print(scoring, scores.mean().round(3), scores.std().round(3))

My five-fold results were an MAE near 47.1, an RMSE near 57.9, and an R² near 0.42. Those sit close to the single-split numbers, which suggests the test split was not a fluke. Remember to flip the sign on the negative scores when you report them.

Watch Out for Overfitting

Random forests can fit training data very closely. Compare the training R² with the test R², and the gap tells the story.

print(rf.score(X_tr, y_tr))   # 0.925
print(rf.score(X_te, y_te))   # 0.44

My forest scored 0.925 on training data and only 0.44 on test data. That large gap shows the forest memorises much of its training set. Never report training metrics as your results.

A handy shortcut exists too. Set oob_score=True, and scikit-learn reports an out-of-bag R² without a separate split. On my run, it landed at 0.442, right next to the test score.

Measurement	R²
Training set	0.925
Test split	0.44
Out-of-bag	0.442

Which Metric Should You Use?

Pick the metric that matches your goal:

1. Use MAE when you want an easy, robust measure and every error costs about the same.
2. Use RMSE when large errors hurt far more than small ones, such as in forecasting or safety work.
3. Use R² when you want to know how much variation your model explains, or when you compare models on the same data.

Report at least two of them. MAE tells the plain story, and RMSE flags the outliers. R² then adds the comparison against a baseline.

Common Mistakes to Avoid

I see these mistakes again and again:

1. Comparing R² across datasets. R² depends on the data's variance, so scores from different datasets do not compare fairly.
2. Ignoring units. MAE and RMSE use the target's units. An RMSE of 54 means something different for prices and for temperatures.
3. Skipping the baseline. A score without a baseline carries no meaning.
4. Reporting training scores. Forests overfit training data, so only test or cross-validated scores count.
5. Forgetting the sign. Scikit-learn's negative scorers flip the sign. Multiply by minus one before you share them.
6. Averaging MAE and RMSE across folds without checking the spread. A mean of 47.1 with a standard deviation of 15 describes a very different model from one with a standard deviation of 2.
7. Quoting R² to four decimal places on 89 test rows. The uncertainty on that estimate is wider than the digits suggest.

Wrapping Up

Three metrics tell three different stories. MAE gives the typical miss, RMSE spotlights the big misses, and R² shows how much the model beats a plain average. Use them together, and always check them against a baseline.

So run your own forest, compute all three, and look at the gaps between them. A wide gap between MAE and RMSE might point to outliers worth investigating. That curiosity usually improves your model more than any hyperparameter tweak.

If you want to know where these numbers come from during training, the out-of-bag score post explains the free validation estimate used above, and the scikit-learn classification recipes apply the same evaluation discipline to classification.

Frequently Asked Questions

Is an R² of 0.44 good for a random forest regressor?

Not on its own. R² only means something against a baseline and against the noise level of your field. Medical and social datasets often carry heavy irreducible noise, so a 0.44 there can be respectable, while a physics dataset with clean measurements would call the same number a failure. Run DummyRegressor first: if the baseline scores near zero, your forest explains 44% of the variance that guessing the mean leaves on the table.

Should I report MAE or RMSE?

Report both, because they answer different questions. MAE is the typical miss in the target's units and is easy to explain; RMSE weights large errors more heavily, so it rises when a few predictions are badly wrong. A large gap between the two, with RMSE well above MAE, is evidence of outliers or a heavy-tailed error distribution worth investigating before you tune anything.

Why is RMSE always greater than or equal to MAE?

Because squaring amplifies larger values before averaging, and the square root does not fully undo that amplification for spread-out errors. The two are only equal when every error has the same magnitude, which is exactly the case in the ten-prediction example where one model misses by 5 every time. Any variation in error size pushes RMSE above MAE.

Can R² be negative?

Yes. R² compares your squared error against the squared error of always predicting the mean. If your model is worse than that, the ratio exceeds one and the score drops below zero. A negative R² on test data usually means a bug, a badly overfit model, or a target whose test-set mean differs sharply from the training mean.

Why are scikit-learn's cross-validation scores negative?

Scikit-learn's scoring API follows a higher-is-better convention, so error metrics are negated: neg_mean_absolute_error and neg_root_mean_squared_error. Cross-validated numbers must be multiplied by minus one before you report them, otherwise an MAE of 47.1 appears as -47.1 and any automated comparison picks the worst model.

What does a large train-to-test R² gap mean?

It means the forest memorised training rows rather than learning a general rule. A gap from 0.925 on training data to 0.44 on test data is severe for a random forest and points at too much depth, too few trees, or leakage between the splits. Compare the out-of-bag score too: if OOB sits near the test score, the split was fair and the model is simply overfitting.

Sources

• [1] scikit-learn, "Metrics and scoring: regression metrics" — the definitions of MAE, MSE, RMSE and R², and the note on why R² can be negative: scikit-learn.org/modules/model_evaluation
• [2] scikit-learn, r2_score API reference — the coefficient of determination, the multioutput handling, and the force_finite behaviour for constant targets: scikit-learn.org/.../r2_score
• [3] scikit-learn, DummyRegressor API reference — the mean, median, quantile and constant strategies that form the baseline this post compares against: scikit-learn.org/.../DummyRegressor
• [4] scikit-learn, cross_val_score API reference — the scoring strings, including the negated error scorers, and the cv argument used for the five-fold run: scikit-learn.org/.../cross_val_score
• [5] scikit-learn, diabetes dataset — the ten baseline variables and the disease-progression target behind every number in this post: scikit-learn.org/.../load_diabetes

Related Articles

• Random Forest OOB Score Explained: Validation Without a Test Set
• Scikit-learn Classification Recipes: Raw Data to Evaluation
• How to Extract Decision Rules from a Random Forest