Sam Austinon October 31, 202617 min read

Lag Features in Pandas: Give Your Random Forest Memory

Lag Features in Pandas: Give Your Random Forest Memory
Contents

Figure 1: Lag Features in Pandas: Give Your Random Forest Memory — trees have no memory of yesterday until you put it in the row

A random forest has no memory. Hand it a row of features, and it predicts from that row alone. It knows nothing about yesterday unless you tell it.

Lag features fix that. This guide shows you how to build them in pandas, feed them to a random forest, and avoid the traps. I ran every experiment on synthetic daily data, and one result changed how I use random forests for time series.

What a Lag Feature Is

A lag feature is a past value of a series, placed in the same row as today's prediction target. The one-day lag of today's sales equals yesterday's sales. The seven-day lag equals last week's value.

In pandas, shift builds them [1]:

df['lag_1'] = df['sales'].shift(1)    # yesterday
df['lag_7'] = df['sales'].shift(7)    # same weekday last week

shift(1) pushes every value down one row, so each row now carries the previous row's value. The first row gets NaN, because it has no past. That simple move turns a sequence into a table that a tabular model can learn from.

Set Up a Test Series

I generated 1,200 days of sales with a gentle upward trend, a weekly cycle, and autocorrelated noise. Then I wrote a helper that builds features on demand.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(0)
n = 1200
idx = pd.date_range('2021-01-01', periods=n, freq='D')
t = np.arange(n)

ar = np.zeros(n)
for i in range(1, n):
    ar[i] = 0.7 * ar[i - 1] + rng.normal(0, 2.0)

y = 100 + 0.08 * t + 10 * np.sin(2 * np.pi * t / 7) + ar
df = pd.DataFrame({'sales': y}, index=idx)

def make(df, lags=(), rolls=(), cal=True):
    d = pd.DataFrame(index=df.index)
    if cal:
        d['dow'] = df.index.dayofweek
        d['month'] = df.index.month
    for L in lags:
        d[f'lag_{L}'] = df['sales'].shift(L)
    for w in rolls:
        d[f'roll_{w}_mean'] = df['sales'].shift(1).rolling(w).mean()
    d['y'] = df['sales']
    return d

def evalrf(d, label, split=0.8):
    d = d.dropna()
    cut = int(len(d) * split)
    X, target = d.drop(columns='y'), d['y']
    rf = RandomForestRegressor(n_estimators=300, random_state=0,
                               n_jobs=-1, min_samples_leaf=2)
    rf.fit(X.iloc[:cut], target.iloc[:cut])
    pred = rf.predict(X.iloc[cut:])
    print(label, mean_absolute_error(target.iloc[cut:], pred))
    return rf, X.columns

Notice the chronological split: the first 80% trains, and the last 20% tests. A random split would leak the future, and our earlier guide covers why.

Experiment 1: How Much Do Lags Help?

I added features step by step and measured mean absolute error (MAE) on the test period.

Features Test MAE
Calendar only (day of week, month) 49.73
Plus lag 1 9.39
Plus lags 1 and 7 8.33
Plus lags 1, 2, 7 and 14 8.14
Plus rolling means (7 and 28 days) 7.69

The first lag does the heavy lifting. Error dropped from about 50 to about 9, because yesterday's value carries the level of the series. Each additional feature then adds a smaller gain. The calendar-only model fails because it knows the weekday pattern but not where the series currently sits.

So far, the story sounds great. Now the plot twist.

The Twist: A Naive Baseline Beats the Forest

Before you celebrate, check a naive baseline. I measured two on the same test period:

Baseline Test MAE
Predict yesterday's value 5.72
Predict last week's value 3.02

The simplest possible forecast, "same weekday last week," beat every random forest above by a wide margin. The best forest reached 7.69, more than twice as bad. If I had skipped the baseline, I would have shipped a model that loses to a one-line formula.

Why did the forest lose? Look at the feature importances for the richest model: lag 7 carried 72% of the importance, lag 14 carried another 26%, and every other feature shared less than 2%. The forest leaned on one number, but it could not use it well, because of a limitation that every tree-based model shares.

A tree predicts by averaging training values in a leaf. It can never output a value outside the range it saw in training [3]. My series trends upward, so the test period contains values higher than anything in training.

The numbers show it. The training maximum was 185.5, the test maximum was 208.5, and the forest's highest prediction was only 183.3. It cannot extrapolate the trend, so its errors grow as the series climbs.

The fix is elegant: predict the change, not the level. Instead of asking the forest for tomorrow's sales, ask it how much sales will differ from a lag, then add the lag back.

d = make(df, lags=(1, 7), rolls=(7,)).dropna()
cut = int(len(d) * 0.8)
X, target = d.drop(columns='y'), d['y']

change = target - X['lag_1']                 # what the model learns

rf = RandomForestRegressor(n_estimators=300, random_state=0,
                           n_jobs=-1, min_samples_leaf=2)
rf.fit(X.iloc[:cut], change.iloc[:cut])

pred = rf.predict(X.iloc[cut:]) + X['lag_1'].iloc[cut:]   # add the level back

The changes stay within a stable range even when the level keeps rising, so the forest has no extrapolation problem. The results:

Target Test MAE
Raw level 7.87
Change from lag 1 2.14
Naive last-week baseline 2.98

Now the forest beats both baselines. Predicting changes cut the error by about 73%.

A Stronger Trend Makes It Dramatic

I also tried a series with a much steeper trend, 0.5 per day instead of 0.08. The raw-target forest scored an MAE of 59.51, which is awful, while the change-from-lag target scored 1.92. The naive yesterday baseline scored 5.73. A steeper trend punishes level prediction harder, and it barely affects change prediction.

This single trick matters more than any other tuning for trending series. If your target drifts upward or downward, use a differenced target, or detrend first.

Build Better Lag Features

Once the target problem is handled, lag features offer plenty of room. Here are the most useful kinds.

Multiple Lags

Pick lags that match the rhythms in your data:

  • Daily data: 1, 2, 7, 14, 28.
  • Hourly data: 1, 24, 168 (one day and one week).
  • Weekly data: 1, 2, 4, 52.

Plot the autocorrelation of your series to find the lags that matter, and test the candidates with a validation set.

Rolling Statistics

Rolling means and standard deviations summarize recent behaviour. Always shift before you roll.

past = df['sales'].shift(1)
df['roll_7_mean'] = past.rolling(7).mean()
df['roll_28_mean'] = past.rolling(28).mean()
df['roll_7_std'] = past.rolling(7).std()

The shift(1) makes each window end at yesterday. Without it, the window includes today's value, which is the very number you want to predict.

Differences and Percent Changes

Differences between lags capture momentum:

df['diff_1'] = df['sales'].shift(1) - df['sales'].shift(2)
df['pct_change_7'] = df['sales'].shift(1) / df['sales'].shift(8) - 1

Both features use only past values. Percent changes also work well when you mix series of different sizes.

Lags for Many Series: Use groupby

Multiple stores, products or sensors stacked in one table need one rule: shift within each group. A plain shift runs across the whole column, so one series' last value leaks into the next series' first row [2].

I tested three stores stacked in one frame. For the first row of each store:

  • Plain shift(1): gave NaN, 37.3 and 154.5, which are other stores' final values.
  • groupby('store').shift(1): gave NaN for all three, which is correct.
panel['lag_1'] = panel.groupby('store')['sales'].shift(1)

panel['roll_7'] = (panel.groupby('store')['sales']
                        .transform(lambda s: s.shift(1).rolling(7).mean()))

Sort by store and date first, so rows sit in time order inside each group. That plain-shift bug is silent, since it only affects the first few rows, and it contaminates the series with another entity's data.

Forecasting Further Ahead

So far, each prediction used yesterday's value, which works for one-step-ahead forecasts. If you need to predict seven days ahead, yesterday's value does not exist yet at prediction time. You have two strategies.

Direct Forecasting

Train a model whose target sits seven days out, and restrict the features to what you will know at the forecast origin.

h = 7
dd = pd.DataFrame({
    'dow_t': (df.index + pd.Timedelta(days=h)).dayofweek,   # weekday of the target day
    'lag_0': df['sales'],
    'lag_1': df['sales'].shift(1),
    'lag_6': df['sales'].shift(6),
    'roll_7': df['sales'].rolling(7).mean(),
    'y_h': df['sales'].shift(-h),
}, index=df.index).dropna()

Here the features at row t describe the world as of day t, and the target is the value at t + 7. Predict the change y_h - lag_0, then add lag_0 back.

Recursive Forecasting

Train a one-step model, then feed each prediction back in as the next step's lag. It needs only one model, but errors can compound along the way.

What Happened When I Tested Both

The seven-day horizon humbled the forest:

Forecast Test MAE
Direct, raw target 8.40
Direct, change target 4.05
Naive same weekday last week 2.99

The change target helped again, cutting the direct error roughly in half. But the naive baseline still won. This series has a strong weekly cycle and noise that forgets itself within a few days, so by day seven nothing in the recent data beats "same weekday last week."

Recursive forecasting was worse than it looks. Error by step, averaged over every seventh origin:

Step ahead Recursive MAE Naive MAE
1 2.65 2.88
2 3.47 3.14
3 5.82 3.52
4 6.11 2.44
5 5.41 2.72
6 5.70 3.32
7 6.94 3.00

At step one the recursive model matched the baseline, and from step three it lost badly. Each prediction became the next prediction's input, so the errors fed on themselves.

That lesson generalises. At short horizons, recent lags matter most. At long horizons, seasonality and trend dominate, and the baseline gets hard to beat.

Practical Details

A few details save you debugging time:

  • NaN rows: each lag creates missing values at the top. The largest lag decides how many rows you lose, so a lag of 28 drops 28 rows. Use dropna(), or choose a model that handles missing values natively, such as HistGradientBoostingRegressor.
  • Regular frequency: lags assume each row equals one time step. Use asfreq to insert missing dates first, or a lag of 7 will not mean one week.
  • Time zones: make sure daily boundaries match your business day before you shift.
  • Known future inputs: calendar fields, planned promotions and holidays can describe the target day without leaking anything.
  • Scaling: random forests do not need scaled features, so skip it here.

A Checklist for Lag Features

Run through this list for every project:

  1. Sort and regularise the index before you shift anything.
  2. Use groupby when the table holds several series.
  3. Shift before rolling, so windows end in the past.
  4. Keep every lag at least as long as the forecast horizon.
  5. Predict changes when the series trends.
  6. Split chronologically, and validate with walk-forward folds.
  7. Compare against naive and seasonal-naive baselines.
  8. Retrain on all history before you deploy.

Common Pitfalls

  1. Predicting the raw level of a trending series with a forest. It cannot extrapolate.
  2. Skipping the baseline. My richest forest lost to a one-line formula until I changed the target.
  3. Using a plain shift on stacked series. The first rows of each group get contaminated.
  4. Rolling without shifting. The window includes the answer.
  5. Using lag 1 for a seven-day forecast. That value will not exist at prediction time.
  6. Adding every lag you can think of. Extra lags add noise, and the gains flatten quickly.
  7. Evaluating with a shuffled split. It leaks the future.

Wrapping Up

Lag features give a random forest the memory it lacks. In my tests, one lag cut the error from about 50 to about 9, and more lags and rolling means trimmed it further. The real jump came from predicting the change from a lag instead of the level, which dropped error to 2.14 and beat the naive baselines.

So build a lag-1 and lag-7 feature set, switch your target to a change from the most recent lag, and compare against "same as last week." If your forest still loses to that baseline, your lags have nothing new to say, and you have saved yourself a deployment.

Frequently Asked Questions

Why do lag features help a random forest?

A random forest predicts each row independently, so it has no memory of previous rows. A lag feature copies a past value into the current row, which gives the model the level and momentum of the series. In my test, adding a single lag cut the error from about 50 to about 9.

Why did my random forest lose to a naive forecast?

Two common reasons. The model may be predicting the raw level of a trending series, which trees cannot extrapolate beyond their training range, or the recent past may simply carry no information the seasonal pattern lacks. Check a seasonal-naive baseline before you tune anything.

Should I predict the level or the change?

For a trending series, predict the change from a recent lag and add the lag back. The changes stay in a stable range even when the level climbs, so the forest never has to extrapolate. In my runs this cut the error from 7.87 to 2.14 on the moderate trend and from 59.51 to 1.92 on the steep one.

How many lags should I use?

Start with the lags that match the rhythms in your data: 1, 2, 7, 14 and 28 for daily data, or 1, 24 and 168 for hourly data. Then plot the autocorrelation and test candidates on a validation set. Extra lags add noise, and the gains flatten quickly after the first few.

How do I create lag features for multiple series?

Use groupby so the shift happens inside each series: panel.groupby('store')['sales'].shift(1). A plain shift runs across the whole column, so the first row of each store receives the previous store's last value. Sort by store and date first so the rows are in time order within each group.

Sources

What are You Looking For?

esc