Contents
Figure 1: Lag Features in Pandas: Give Your Random Forest Memory — trees have no memory of yesterday until you put it in the row
A random forest has no memory. Hand it a row of features, and it predicts from that row alone. It knows nothing about yesterday unless you tell it.
Lag features fix that. This guide shows you how to build them in pandas, feed them to a random forest, and avoid the traps. I ran every experiment on synthetic daily data, and one result changed how I use random forests for time series.
What a Lag Feature Is
A lag feature is a past value of a series, placed in the same row as today's prediction target. The one-day lag of today's sales equals yesterday's sales. The seven-day lag equals last week's value.
In pandas, shift builds them [1]:
df['lag_1'] = df['sales'].shift(1) # yesterday
df['lag_7'] = df['sales'].shift(7) # same weekday last week
shift(1) pushes every value down one row, so each row now carries the previous row's value. The first row gets NaN, because it has no past. That simple move turns a sequence into a table that a tabular model can learn from.
Set Up a Test Series
I generated 1,200 days of sales with a gentle upward trend, a weekly cycle, and autocorrelated noise. Then I wrote a helper that builds features on demand.
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
rng = np.random.default_rng(0)
n = 1200
idx = pd.date_range('2021-01-01', periods=n, freq='D')
t = np.arange(n)
ar = np.zeros(n)
for i in range(1, n):
ar[i] = 0.7 * ar[i - 1] + rng.normal(0, 2.0)
y = 100 + 0.08 * t + 10 * np.sin(2 * np.pi * t / 7) + ar
df = pd.DataFrame({'sales': y}, index=idx)
def make(df, lags=(), rolls=(), cal=True):
d = pd.DataFrame(index=df.index)
if cal:
d['dow'] = df.index.dayofweek
d['month'] = df.index.month
for L in lags:
d[f'lag_{L}'] = df['sales'].shift(L)
for w in rolls:
d[f'roll_{w}_mean'] = df['sales'].shift(1).rolling(w).mean()
d['y'] = df['sales']
return d
def evalrf(d, label, split=0.8):
d = d.dropna()
cut = int(len(d) * split)
X, target = d.drop(columns='y'), d['y']
rf = RandomForestRegressor(n_estimators=300, random_state=0,
n_jobs=-1, min_samples_leaf=2)
rf.fit(X.iloc[:cut], target.iloc[:cut])
pred = rf.predict(X.iloc[cut:])
print(label, mean_absolute_error(target.iloc[cut:], pred))
return rf, X.columns
Notice the chronological split: the first 80% trains, and the last 20% tests. A random split would leak the future, and our earlier guide covers why.
Experiment 1: How Much Do Lags Help?
I added features step by step and measured mean absolute error (MAE) on the test period.
| Features | Test MAE |
|---|---|
| Calendar only (day of week, month) | 49.73 |
| Plus lag 1 | 9.39 |
| Plus lags 1 and 7 | 8.33 |
| Plus lags 1, 2, 7 and 14 | 8.14 |
| Plus rolling means (7 and 28 days) | 7.69 |
The first lag does the heavy lifting. Error dropped from about 50 to about 9, because yesterday's value carries the level of the series. Each additional feature then adds a smaller gain. The calendar-only model fails because it knows the weekday pattern but not where the series currently sits.
So far, the story sounds great. Now the plot twist.
The Twist: A Naive Baseline Beats the Forest
Before you celebrate, check a naive baseline. I measured two on the same test period:
| Baseline | Test MAE |
|---|---|
| Predict yesterday's value | 5.72 |
| Predict last week's value | 3.02 |
The simplest possible forecast, "same weekday last week," beat every random forest above by a wide margin. The best forest reached 7.69, more than twice as bad. If I had skipped the baseline, I would have shipped a model that loses to a one-line formula.
Why did the forest lose? Look at the feature importances for the richest model: lag 7 carried 72% of the importance, lag 14 carried another 26%, and every other feature shared less than 2%. The forest leaned on one number, but it could not use it well, because of a limitation that every tree-based model shares.
Why Random Forests Fail at Trends
A tree predicts by averaging training values in a leaf. It can never output a value outside the range it saw in training [3]. My series trends upward, so the test period contains values higher than anything in training.
The numbers show it. The training maximum was 185.5, the test maximum was 208.5, and the forest's highest prediction was only 183.3. It cannot extrapolate the trend, so its errors grow as the series climbs.
The fix is elegant: predict the change, not the level. Instead of asking the forest for tomorrow's sales, ask it how much sales will differ from a lag, then add the lag back.
d = make(df, lags=(1, 7), rolls=(7,)).dropna()
cut = int(len(d) * 0.8)
X, target = d.drop(columns='y'), d['y']
change = target - X['lag_1'] # what the model learns
rf = RandomForestRegressor(n_estimators=300, random_state=0,
n_jobs=-1, min_samples_leaf=2)
rf.fit(X.iloc[:cut], change.iloc[:cut])
pred = rf.predict(X.iloc[cut:]) + X['lag_1'].iloc[cut:] # add the level back
The changes stay within a stable range even when the level keeps rising, so the forest has no extrapolation problem. The results:
| Target | Test MAE |
|---|---|
| Raw level | 7.87 |
| Change from lag 1 | 2.14 |
| Naive last-week baseline | 2.98 |
Now the forest beats both baselines. Predicting changes cut the error by about 73%.
A Stronger Trend Makes It Dramatic
I also tried a series with a much steeper trend, 0.5 per day instead of 0.08. The raw-target forest scored an MAE of 59.51, which is awful, while the change-from-lag target scored 1.92. The naive yesterday baseline scored 5.73. A steeper trend punishes level prediction harder, and it barely affects change prediction.
This single trick matters more than any other tuning for trending series. If your target drifts upward or downward, use a differenced target, or detrend first.
Build Better Lag Features
Once the target problem is handled, lag features offer plenty of room. Here are the most useful kinds.
Multiple Lags
Pick lags that match the rhythms in your data:
- Daily data: 1, 2, 7, 14, 28.
- Hourly data: 1, 24, 168 (one day and one week).
- Weekly data: 1, 2, 4, 52.
Plot the autocorrelation of your series to find the lags that matter, and test the candidates with a validation set.
Rolling Statistics
Rolling means and standard deviations summarize recent behaviour. Always shift before you roll.
past = df['sales'].shift(1)
df['roll_7_mean'] = past.rolling(7).mean()
df['roll_28_mean'] = past.rolling(28).mean()
df['roll_7_std'] = past.rolling(7).std()
The shift(1) makes each window end at yesterday. Without it, the window includes today's value, which is the very number you want to predict.
Differences and Percent Changes
Differences between lags capture momentum:
df['diff_1'] = df['sales'].shift(1) - df['sales'].shift(2)
df['pct_change_7'] = df['sales'].shift(1) / df['sales'].shift(8) - 1
Both features use only past values. Percent changes also work well when you mix series of different sizes.
Lags for Many Series: Use groupby
Multiple stores, products or sensors stacked in one table need one rule: shift within each group. A plain shift runs across the whole column, so one series' last value leaks into the next series' first row [2].
I tested three stores stacked in one frame. For the first row of each store:
- Plain
shift(1): gave NaN, 37.3 and 154.5, which are other stores' final values. groupby('store').shift(1): gave NaN for all three, which is correct.
panel['lag_1'] = panel.groupby('store')['sales'].shift(1)
panel['roll_7'] = (panel.groupby('store')['sales']
.transform(lambda s: s.shift(1).rolling(7).mean()))
Sort by store and date first, so rows sit in time order inside each group. That plain-shift bug is silent, since it only affects the first few rows, and it contaminates the series with another entity's data.
Forecasting Further Ahead
So far, each prediction used yesterday's value, which works for one-step-ahead forecasts. If you need to predict seven days ahead, yesterday's value does not exist yet at prediction time. You have two strategies.
Direct Forecasting
Train a model whose target sits seven days out, and restrict the features to what you will know at the forecast origin.
h = 7
dd = pd.DataFrame({
'dow_t': (df.index + pd.Timedelta(days=h)).dayofweek, # weekday of the target day
'lag_0': df['sales'],
'lag_1': df['sales'].shift(1),
'lag_6': df['sales'].shift(6),
'roll_7': df['sales'].rolling(7).mean(),
'y_h': df['sales'].shift(-h),
}, index=df.index).dropna()
Here the features at row t describe the world as of day t, and the target is the value at t + 7. Predict the change y_h - lag_0, then add lag_0 back.
Recursive Forecasting
Train a one-step model, then feed each prediction back in as the next step's lag. It needs only one model, but errors can compound along the way.
What Happened When I Tested Both
The seven-day horizon humbled the forest:
| Forecast | Test MAE |
|---|---|
| Direct, raw target | 8.40 |
| Direct, change target | 4.05 |
| Naive same weekday last week | 2.99 |
The change target helped again, cutting the direct error roughly in half. But the naive baseline still won. This series has a strong weekly cycle and noise that forgets itself within a few days, so by day seven nothing in the recent data beats "same weekday last week."
Recursive forecasting was worse than it looks. Error by step, averaged over every seventh origin:
| Step ahead | Recursive MAE | Naive MAE |
|---|---|---|
| 1 | 2.65 | 2.88 |
| 2 | 3.47 | 3.14 |
| 3 | 5.82 | 3.52 |
| 4 | 6.11 | 2.44 |
| 5 | 5.41 | 2.72 |
| 6 | 5.70 | 3.32 |
| 7 | 6.94 | 3.00 |
At step one the recursive model matched the baseline, and from step three it lost badly. Each prediction became the next prediction's input, so the errors fed on themselves.
That lesson generalises. At short horizons, recent lags matter most. At long horizons, seasonality and trend dominate, and the baseline gets hard to beat.
Practical Details
A few details save you debugging time:
- NaN rows: each lag creates missing values at the top. The largest lag decides how many rows you lose, so a lag of 28 drops 28 rows. Use
dropna(), or choose a model that handles missing values natively, such asHistGradientBoostingRegressor. - Regular frequency: lags assume each row equals one time step. Use
asfreqto insert missing dates first, or a lag of 7 will not mean one week. - Time zones: make sure daily boundaries match your business day before you shift.
- Known future inputs: calendar fields, planned promotions and holidays can describe the target day without leaking anything.
- Scaling: random forests do not need scaled features, so skip it here.
A Checklist for Lag Features
Run through this list for every project:
- Sort and regularise the index before you shift anything.
- Use
groupbywhen the table holds several series. - Shift before rolling, so windows end in the past.
- Keep every lag at least as long as the forecast horizon.
- Predict changes when the series trends.
- Split chronologically, and validate with walk-forward folds.
- Compare against naive and seasonal-naive baselines.
- Retrain on all history before you deploy.
Common Pitfalls
- Predicting the raw level of a trending series with a forest. It cannot extrapolate.
- Skipping the baseline. My richest forest lost to a one-line formula until I changed the target.
- Using a plain
shifton stacked series. The first rows of each group get contaminated. - Rolling without shifting. The window includes the answer.
- Using lag 1 for a seven-day forecast. That value will not exist at prediction time.
- Adding every lag you can think of. Extra lags add noise, and the gains flatten quickly.
- Evaluating with a shuffled split. It leaks the future.
Wrapping Up
Lag features give a random forest the memory it lacks. In my tests, one lag cut the error from about 50 to about 9, and more lags and rolling means trimmed it further. The real jump came from predicting the change from a lag instead of the level, which dropped error to 2.14 and beat the naive baselines.
So build a lag-1 and lag-7 feature set, switch your target to a change from the most recent lag, and compare against "same as last week." If your forest still loses to that baseline, your lags have nothing new to say, and you have saved yourself a deployment.
Frequently Asked Questions
Why do lag features help a random forest?
A random forest predicts each row independently, so it has no memory of previous rows. A lag feature copies a past value into the current row, which gives the model the level and momentum of the series. In my test, adding a single lag cut the error from about 50 to about 9.
Why did my random forest lose to a naive forecast?
Two common reasons. The model may be predicting the raw level of a trending series, which trees cannot extrapolate beyond their training range, or the recent past may simply carry no information the seasonal pattern lacks. Check a seasonal-naive baseline before you tune anything.
Should I predict the level or the change?
For a trending series, predict the change from a recent lag and add the lag back. The changes stay in a stable range even when the level climbs, so the forest never has to extrapolate. In my runs this cut the error from 7.87 to 2.14 on the moderate trend and from 59.51 to 1.92 on the steep one.
How many lags should I use?
Start with the lags that match the rhythms in your data: 1, 2, 7, 14 and 28 for daily data, or 1, 24 and 168 for hourly data. Then plot the autocorrelation and test candidates on a validation set. Extra lags add noise, and the gains flatten quickly after the first few.
How do I create lag features for multiple series?
Use groupby so the shift happens inside each series: panel.groupby('store')['sales'].shift(1). A plain shift runs across the whole column, so the first row of each store receives the previous store's last value. Sort by store and date first so the rows are in time order within each group.
Sources
[1]pandas, DataFrame.shift — the method behind every lag feature, with theperiodsandfreqarguments and the warning thatfreqshould not be used when the index has gaps: pandas.pydata.org/docs/reference/api/pandas.DataFrame.shift.html[2]pandas, Group by: split-apply-combine — the user guide forgroupby, including groupedshiftandtransform, which keep each series' history inside its own group: pandas.pydata.org/docs/user_guide/groupby.html[3]scikit-learn, Decision Trees — the user guide statement that tree predictions are piecewise constant approximations and "therefore, they are not good at extrapolation": scikit-learn.org/stable/modules/tree.html[4]scikit-learn, RandomForestRegressor — the API reference forn_estimators,min_samples_leafandfeature_importances_, the three settings used in every experiment here: scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html[5]Hyndman and Athanasopoulos, Forecasting: Principles and Practice, lagged predictors — the textbook treatment of distributed lags, where the target depends on a predictor and its lagsx_{t-1}throughx_{t-k}, with the lag lengthkchosen from the data: otexts.com/fpp3/lagged-predictors.html