Contents
Figure 1: How to Split Time Series Data Without Leaking the Future — the score that survives production is the one measured on data the model has never seen
Your forecasting model scores beautifully in validation. Then it goes live, and its error doubles. More often than not the culprit is leakage: information from the future slipped into training, and the model learned to cheat.
This guide shows you how to split time series data for machine learning without leaking the future. We cover chronological splits, TimeSeriesSplit, safe features, preprocessing, gaps and tuning. I ran every experiment on synthetic data, including a few results that complicate the usual advice.
What "Leaking the Future" Means
A time series model must predict tomorrow using only what you know today. Leakage happens whenever training or validation uses information that would not exist at prediction time. It sneaks in through four doors:
| Door | What it looks like | Where it bites |
|---|---|---|
| The split | Random shuffling puts future rows in training and past rows in testing | Every score you report |
| The features | A rolling average that includes tomorrow's value | The model's inputs |
| The preprocessing | Scaling or imputing on the full dataset | Every fold of every run |
| Overlapping labels | A target that spans several days, overlapping across the boundary | The rows near the cut |
We will close each door in order. First, let's build some data.
Set Up a Test Series
I generated 1,500 days of data with a gentle upward trend, a weekly cycle, and autocorrelated noise. Then I built lag and rolling features.
import numpy as np
import pandas as pd
rng = np.random.default_rng(0)
n = 1500
t = np.arange(n)
ar = np.zeros(n)
for i in range(1, n):
ar[i] = 0.85 * ar[i - 1] + rng.normal(0, 1.0)
y = 50 + 0.05 * t + 8 * np.sin(2 * np.pi * t / 7) + 4 * ar
df = pd.DataFrame({'y': y}, index=pd.date_range('2020-01-01', periods=n, freq='D'))
def build(df):
d = pd.DataFrame(index=df.index)
d['lag1'] = df.y.shift(1)
d['lag2'] = df.y.shift(2)
d['lag7'] = df.y.shift(7)
d['dow'] = df.index.dayofweek
d['roll7'] = df.y.shift(1).rolling(7).mean() # past values only
d['y'] = df.y
return d.dropna()
data = build(df)
X, target = data.drop(columns='y'), data['y']
Pay attention to shift(1) before rolling(7). That order makes the rolling average use only days before the prediction day. We will break it on purpose later.
Rule 1: Split by Time, Not at Random
The simplest honest split holds out the last chunk of data. Train on the past, test on the future, exactly as production will.
cut = int(len(data) * 0.8)
X_train, X_test = X.iloc[:cut], X.iloc[cut:]
y_train, y_test = target.iloc[:cut], target.iloc[cut:]
Now compare it with a random 80/20 split. I trained a random forest on each, using the same features, and measured mean absolute error (MAE) on the test rows.
| Split | Test MAE |
|---|---|
| Chronological holdout | 4.43 |
| Random 80/20 | 4.05 |
The random split looked about 9% better than reality. That gap is real but modest, and I want to be honest about it. Here is why it happens: each test row's value appears as a lag feature in the next training row, so the model effectively sees the answers in its inputs.
When the Gap Gets Huge
The damage grows when a series has a trend and your features include time itself. I trained a forest on only the day number and weekday:
| Split | Test MAE |
|---|---|
| Chronological holdout | 11.92 |
| Random 80/20 | 4.23 |
The random split understated the real error by nearly a factor of three. Tree models cannot extrapolate a trend beyond the range they trained on, so the future test exposes them, while the shuffled test lets them interpolate between neighbours.
When the Gap Shrinks
I also tested a persistent series, with strong autocorrelation and lag features only. There the random split scored an MAE of 3.36 and the chronological holdout scored 3.78, so the flattery fell to about 11%.
I then swept the persistence level and it moved again: between about 7% and 24% depending on how autocorrelated the series was. In every case the random split looked better than reality, never worse. So the size of the lie is not something you can predict from the data in front of you, and that unpredictability is the whole argument for a split that mimics production.
Rule 2: Cross-Validate With TimeSeriesSplit
A single holdout gives you one number. Cross-validation gives you several, but ordinary shuffled K-fold repeats the random-split mistake. Scikit-learn's TimeSeriesSplit fixes that: each fold trains on the past and tests on the block right after it [1][2].
from sklearn.model_selection import TimeSeriesSplit, KFold, cross_val_score
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_estimators=200, random_state=0, n_jobs=-1)
shuffled = KFold(5, shuffle=True, random_state=0)
forward = TimeSeriesSplit(n_splits=5)
mae_kf = -cross_val_score(model, X_train, y_train, cv=shuffled,
scoring='neg_mean_absolute_error').mean()
mae_ts = -cross_val_score(model, X_train, y_train, cv=forward,
scoring='neg_mean_absolute_error').mean()
The results on my data:
| Estimate | MAE |
|---|---|
| Shuffled K-fold CV | 3.79 |
| TimeSeriesSplit CV | 4.84 |
| True future holdout | 4.43 |
Shuffled CV flattered the model by about 14%, while the forward-chaining CV came out slightly pessimistic, which is the safer direction. It also costs you something: early folds train on little data.
See the Fold Structure
To understand the splits, look at the indexes on a tiny example of 20 rows with 3 folds:
| Fold | Training rows | Test rows |
|---|---|---|
| 0 | 0 to 4 | 5 to 9 |
| 1 | 0 to 9 | 10 to 14 |
| 2 | 0 to 14 | 15 to 19 |
The training window expands. Each fold adds the previous test block to training, which mimics retraining as new data arrives [4].
Three Useful Options
The TimeSeriesSplit class takes arguments that matter in practice [1]:
test_size: fixes the test block length, so every fold tests the same horizon.gap: skips rows between training and testing. Withgap=2andtest_size=4, my tiny example trained on rows 0 to 5 and tested on rows 8 to 11.max_train_size: caps the training window, which turns the expanding window into a sliding window. Withmax_train_size=6, the folds trained on rows 2 to 7, then 6 to 11, then 10 to 15.
Choose a sliding window when old data stops being relevant, such as after a regime change. Choose an expanding window when older history still helps.
Rule 3: Build Features From the Past Only
A clean split cannot save you from leaky features. The classic mistake uses a centered rolling window, which averages values on both sides of each row, including the row itself.
# LEAKY: includes the current value and future values
d['roll7_leaky'] = df.y.rolling(7, center=True).mean()
# SAFE: shift first, so only past values enter the window
d['roll7'] = df.y.shift(1).rolling(7).mean()
I added the leaky feature and re-ran the chronological holdout. The error fell from 4.43 to 3.88, even though the split was honest. That improvement is fake, because the feature uses information the model cannot have at prediction time. In my noisy synthetic series the effect stayed modest; with cleaner signals it can become dramatic.
Here is a checklist for every feature:
- Ask: would I know this value at the moment I need the prediction?
- Shift before you roll. Apply
shift(1)first, thenrollingorexpandingwindows. - Watch grouped aggregates. A per-customer mean computed over the whole dataset includes the future.
- Check backward fills.
bfillcopies future values backward in time, and forward fill is the safe choice. - Allow known-in-advance features. Holidays, planned promotions and calendar fields are fine, because you know them ahead of time.
Rule 4: Fit Preprocessing on Training Data Only
Scalers, imputers, encoders and feature selectors all learn from data. If they learn from the full dataset, test statistics leak into training [3]. Put them in a pipeline and fit the pipeline on each training fold [5].
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
pipe = make_pipeline(StandardScaler(), Ridge())
pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))
The pipeline refits the scaler inside every cross-validation fold, so each fold learns its statistics from its own past. On my data the ridge pipeline scored a test MAE of 4.02. A pipeline protects you from this whole class of bugs, and it costs three lines.
Also watch for target-based features, like target encoding or the mean of the target by category. Compute them using training rows only, and for each fold, only from earlier rows.
Rule 5: Leave a Gap When Labels or Features Overlap
Suppose you forecast sales 7 days ahead. A training row at day 100 has a label that depends on days up to 107. If your test block starts at day 103, the training label overlaps with test-period information.
The fix is a gap between training and test. Use the gap argument in TimeSeriesSplit, and set it at least as large as your forecast horizon. A common rule of thumb adds the longest lookback window of your features as well. Quant finance calls the same idea "purging" and "embargo."
cv = TimeSeriesSplit(n_splits=5, gap=7, test_size=30)
Without a gap, you leak a little through the boundary. With one, you give up a few rows of training data, which is a cheap price.
Rule 6: Tune Inside Time-Aware CV, and Keep a Final Holdout
Hyperparameter search reuses your validation data many times, which makes the best score optimistic. Do the search with a time-aware splitter, then evaluate once on a final holdout that no search ever touched.
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import mean_absolute_error
search = GridSearchCV(
RandomForestRegressor(random_state=0),
{'max_depth': [5, 10, None], 'min_samples_leaf': [1, 5, 20]},
cv=TimeSeriesSplit(n_splits=5, gap=7),
scoring='neg_mean_absolute_error',
)
search.fit(X_train, y_train)
print(mean_absolute_error(y_test, search.predict(X_test)))
After you finish, retrain on all available data, including the holdout, before you deploy. Production should use the freshest history.
Rule 7: Handle Many Series Carefully
Panel data, such as sales for hundreds of stores, adds a question: what are you generalising to?
- Future dates for the same series: split by a time cutoff that applies to every series. All training rows come before the cutoff, and all test rows come after.
- New, unseen series: split by entity, holding out whole stores, and also respect time if the problem is also temporal.
Mixing the two quietly leaks. For example, if store A's Tuesday sits in training and store B's Monday sits in testing, a shared market event could leak across.
Always Beat a Naive Baseline
A good validation score means little without a baseline. Forecasting has simple ones: repeat yesterday, or repeat the same weekday last week.
| Forecast | Test MAE |
|---|---|
| Yesterday's value | 5.45 |
| Same weekday last week | 7.17 |
| Random forest | 4.43 |
The forest beat both, which gives that number meaning. If a fancy model cannot beat a naive baseline on honest data, you have learned something valuable.
The Leak-Proof Checklist
Run through this list before you trust any time series result:
- Split chronologically, and never shuffle.
- Use
TimeSeriesSplitfor cross-validation, withgapset to your horizon. - Shift before rolling, and avoid centered windows.
- Fit every transformer inside a pipeline on training data only.
- Compute target-based features from earlier rows only.
- Tune with time-aware CV, then test once on a final holdout.
- Compare against naive baselines.
- Retrain on all data before deployment.
Common Pitfalls
- Using
train_test_splitwith defaults. It shuffles, so it leaks. - Scaling the full dataset first. The test statistics sneak into training.
- Using
bfillor interpolation across the split. Both pull future values backward. - Trusting one lucky fold. Average several folds and look at their spread.
- Reporting the best CV score from a big search as your expected performance. It carries optimism.
- Forgetting that trees cannot extrapolate trends. Add detrending, differencing or lag features.
Wrapping Up
Honest time series validation means training on the past, testing on the future, building features from only what you would know, and leaving gaps where labels overlap. In my experiments, shuffled validation flattered the model by about 14% in cross-validation, and a time-index model looked nearly three times better than reality on a random split. Chronological splits never misled in a systematic way.
So take your current forecasting project, and swap its splitter for TimeSeriesSplit with a gap. If your score drops, that is not bad news. You just found out what your model will really do next month.
Frequently Asked Questions
Why can't I use train_test_split on time series?
train_test_split shuffles by default, so future rows land in training and past rows land in testing. The model then interpolates between neighbours it has already seen, and your error estimate is too low. On my synthetic series a random split reported a mean absolute error of 4.05 against a true 4.43, and with a time-index model the gap grew to nearly three times.
How big should the gap be in TimeSeriesSplit?
At least as large as your forecast horizon, because a training row's label can depend on days that fall inside the test block. If your features also look back a long way, add that lookback to the gap. A gap of 7 with a 7-day horizon is the minimum, not a generous setting.
Does TimeSeriesSplit ever overestimate performance?
It can be slightly pessimistic, not optimistic, because early folds train on very little data. In my runs the forward-chaining estimate was 4.84 against a true holdout of 4.43, about 9% worse than reality. That is the safer direction for a number you plan to ship against.
Should I retrain on the holdout before deployment?
Yes, once the evaluation is finished and the configuration is frozen. The holdout exists to give you one honest number, not to stay unused forever. Retraining on all available history gives production the freshest data, but it also means you no longer have an untouched set, so record the holdout score before you do it.
How do I split panel data with many series?
Decide what you are generalising to first. For future dates on the same series, cut every series at the same timestamp. For new unseen series, hold out whole entities. If the problem is both, hold out entities and respect the time cutoff, and never let one store's Tuesday sit in training while another store's Monday sits in testing.
Sources
[1]scikit-learn, TimeSeriesSplit — the API reference forn_splits,max_train_size,test_sizeandgap, described as the cross-validator for time-ordered data where other methods would train on future observations: scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html[2]scikit-learn, Cross-validation: evaluating estimator performance — the user guide section on cross-validation of time series data, including the forward-chaining diagram thatTimeSeriesSplitimplements: scikit-learn.org/stable/modules/cross_validation.html[3]scikit-learn, Common pitfalls and recommended practices — the data leakage section, which defines leakage as using information that would not be available at prediction time and warns that the result is an overly optimistic performance estimate: scikit-learn.org/stable/common_pitfalls.html[4]Hyndman and Athanasopoulos, Forecasting: Principles and Practice, section 5.10 on time series cross-validation — the textbook treatment of rolling-origin evaluation, where the forecast origin moves forward and each evaluation uses only data available at that origin: otexts.com/fpp3/tscv.html[5]scikit-learn, Pipelines and composite estimators — howPipelinechains transformers and a final estimator, and why fitting the whole pipeline keeps each step's statistics inside the training fold: scikit-learn.org/stable/modules/compose.html