Contents
Figure 1: 7 Data Leakage Traps in Time Series ML (and How to Catch Them) — seven ways a feature quietly reads the answer, and the one cheap test that catches four of them
Your model scores beautifully in validation, then falls apart in production. Congratulations: you have met data leakage, the bug that flatters you right up until it matters.
I built a small synthetic test for each of seven common traps. Some leaked badly. One did nothing at all. I will show both, because a trap you cannot measure is a trap you cannot prioritize.
| # | Trap | Leaky vs honest MAE | Detector sees it? |
|---|---|---|---|
| 1 | Window includes today | 0.718 vs 1.110 | yes |
| 2 | Centered window | 0.659 vs 1.110 | yes |
| 3 | Shuffled split | 0.943 vs 3.848 | no |
| 4 | Interpolation across gaps | 0.929 vs 1.068 | only near a gap |
| 5 | Target encoding on all rows | 1.538 vs 4.568 | yes |
| 6 | Overlapping target, no gap | 0.731 vs 0.730 | no |
| 7 | Event-date join | 1.203 vs 2.660 | no |
MAE, lower is better. The rows use different series, models and splits, so compare each row's two numbers, not the rows against each other.
The Detector You Will Want First
Before the traps, here is a check that catches many of them. It is the perturbation test:
- Build your features.
- Change the target series from row
t0onward. I added 1,000. - Rebuild the features and compare every row up to and including
t0. - If any value moved, that feature peeks at the future.
It is cheap and blunt. One lesson from my own run: a single t0 missed the interpolation leak in trap 4, because that leak only fires when the changed row sits in a gap. Test at many start rows. I used 20.
| Feature builder | Start rows that flagged it (of 20) |
|---|---|
rolling(3).mean() |
20 |
rolling(5, center=True).mean() |
20 |
shift(1).rolling(3).mean() |
0 |
| Interpolated lag-1 column | 4 |
| Expanding mean over all rows | 20 |
Four of the twenty start rows caught the interpolation leak. Pick one t0 at random and you have a one-in-five chance of noticing. That is why the sweep matters more than the test itself.
The detector cannot see traps 3, 6 or 7. Those involve splits or timing rather than a feature reading y, and I will flag them as we go.
Trap 1: The Window Includes Today
y.rolling(3).mean() includes the current row by default [1]. If you predict y, your feature contains part of the answer.
The catch: the detector flags it, every time. The fix: y.shift(1).rolling(3).mean().
The cost, in my test: a leaky MAE of 0.718 versus 1.110 for the causal version. The leak made the model look about a third better than it is.
Trap 2: The Centered Window
rolling(5, center=True) labels each window at its middle, so it sees two rows ahead [1]. It is fine for smoothing a chart. It is a leak as a model input.
The catch: detector. The cost: 0.659 versus 1.110 for the causal feature, a 41% discount.
Why do people do it? Plots look nicer. Keep centered windows far from your feature code, or name the function so loudly that nobody reuses it.
Trap 3: The Shuffled Split
Plain shuffled K-fold mixes past and future. How much it hurts depends on what your model can memorize.
I gave a gradient-boosted model a numeric time index and a series with a slowly wandering level. The shuffled score was 0.943. The forward score was 3.848. A tree can look up "what happened near row 1,400" when neighbours from both sides sit in training, but it cannot do that for rows beyond the end.
The catch: compare shuffled and forward scores. A gap this size is an alarm. The fix: TimeSeriesSplit, or any forward-only scheme.
Fair warning: this is a deliberately exposed construction. Pure lag features can show a much smaller gap, and in an earlier test of mine a shuffled split barely flattered the model at all.
Trap 4: Filling Gaps With the Future
Interpolation fills a missing value using points on both sides. If your feature is "yesterday's reading" with holes, the point after a hole is today's target.
I deleted 25% of a lag-1 sensor's readings and compared fills:
| Fill method | MAE |
|---|---|
interpolate() |
0.929 |
ffill() |
1.068 |
Interpolation looked 13% better because it borrowed from tomorrow's row [2]. The catch: detector, but only at several start rows. The fix: forward-fill, or fit a fill rule using only past values.
Trap 5: Encodings Built From All the Rows
Target encoding replaces a category with its average target. Compute that average over the whole dataset and each row's own target helps produce its feature.
The size of this leak depends on how many rows each category has:
| Levels in the ID | Rows per level | Leaky MAE | Causal MAE |
|---|---|---|---|
| 150 | 13.3 | 3.066 | 3.497 |
| 1,000 | 2.0 | 2.206 | 4.274 |
| 2,000 | 1.0 | 1.538 | 4.568 |
With 13 rows per level, the leak was worth 12%. With one row per level, it cut the error by 66%. High-cardinality IDs are the danger: the fewer rows behind a category average, the more of that average is the row's own target.
The catch: detector. The fix: an expanding mean of earlier rows only, shift(1).expanding().mean() per group. Scikit-learn's TargetEncoder cross-fits internally for the same reason, so the encoding a row receives never saw that row's target [5].
Trap 6: Overlapping Targets and the Gap
If your target is "the mean of the next 48 values," rows near the train/test boundary share future data. TimeSeriesSplit has a gap parameter for this, which the docs describe as the number of samples excluded from the end of each training set [3].
I tested it with a Ridge model on two lags. The result: MAE 0.731 with no gap versus 0.730 with a gap of 48. No meaningful difference. My earlier test of the same idea also came out flat, 0.269 against 0.269.
So is the trap fake? I would not say that. My two experiments just did not catch it. I would still add a gap when targets overlap, because it costs little, but I cannot tell you it will change your numbers.
Trap 7: Joining by Event Date Instead of Availability
An indicator for Tuesday might be published on Wednesday. Join it by its own date and your model "knows" Tuesday's value on Tuesday.
I built a daily indicator that is published a day late, with a target that depends on the same-day value:
| Join key | MAE |
|---|---|
| Event date (leaky) | 1.203 |
| Previous day only (available) | 2.660 |
The gap here is by construction, since I made the target depend on the indicator. Real data will vary, but the mechanism is common with revisions, filings and reporting lags.
The catch: the detector cannot see this one, because the indicator does not depend on y. You need timestamps for when each value became known. The fix: join on availability time with pd.merge_asof, which matches backward by default, and add an explicit lag when you lack publish times [4].
A Bonus Non-Finding: The Scaler
Fitting StandardScaler on all rows is a textbook warning. In my run, the fitted mean of one feature was 0.143 on all rows and 0.092 on the first 80%. So the fit does change, by about a third.
I did not score a model with it, so I will not claim how much it matters here. Fit scalers on training data inside a pipeline anyway: it is free.
Your Pre-Flight Checklist
- Run the perturbation test on every feature builder, at many start rows.
- Use forward-only splits. Treat a big shuffled-versus-forward gap as an alarm.
- Shift before you roll.
- Fill gaps with past values only.
- Build encodings from earlier rows.
- Add a gap when targets overlap.
- Join on when data became known, not on its label date.
- Keep a final hold-out you touch once.
Common Pitfalls
- Testing the detector at a single start row. It caught the interpolation leak 4 times out of 20.
- Trusting a good score as evidence of no leak. The leak is invisible in the metric by definition.
- Assuming every leak shows up as a huge gain. Trap 5 at 13 rows per level was worth 12%, easy to wave off.
- Fixing the split and calling it done. A shuffled split was only one of seven traps, and three of them survive a correct split.
- Blaming production for a validation bug. If the score drops on deploy, look at the features before the model.
- Adding a gap and assuming the target overlap is solved. It moves nothing when the leak is in the feature, as traps 1, 2, 4, 5 and 7 show.
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Feature Engineering for Machine Learning | the target-encoding chapter explains, in plain terms, why an aggregate that includes the current row leaks, which is trap 5 here. It also covers the time-based feature traps that make the other six easier to spot in code review. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Wrapping Up
Leakage rarely announces itself. The traps with the biggest measured effects in my tests were the sneaky ones: a window that includes today, an interpolation across a gap, an ID with few rows per level, and a late-published indicator. The one I could not demonstrate, the gap, taught me something too: measure, do not assume.
Pick the model you trust most and run the perturbation test on its feature code today. If a single feature moves, you have just saved yourself a production surprise.
Frequently Asked Questions
What is the fastest way to detect data leakage in time series features?
The perturbation test. Build your features, add a large constant to the target from one row onward, rebuild the features, and compare every row up to that point. If any value moved, the feature read the future. Test at many start rows, because an interpolation leak only fires when the changed row sits next to a gap.
Does pandas rolling include the current row?
Yes. rolling(3).mean() includes the row it is labelled with, because closed=None means the right edge of the window. If the column you are rolling is the target, the feature contains part of the answer. Use y.shift(1).rolling(3).mean() or closed="left" to end the window at the previous row.
Why does interpolation leak in time series?
Because interpolate fills a missing value using points on both sides of the gap. When the missing column is a lag of the target, the fill borrows a future target value. Forward-filling only uses the past. In my test, interpolation scored 0.929 MAE against 1.068 for ffill, and the gain was entirely borrowed from tomorrow.
Is target encoding safe with high-cardinality IDs?
It is most dangerous there. A category with one or two rows gets an encoding that is mostly its own target, so the model can read the answer. In my test, 13 rows per level cut the error by 12%, two rows per level cut it by 48%, and one row per level cut it by 66%. Use an expanding mean of earlier rows, or a cross-fitted encoder.
Does a gap in TimeSeriesSplit stop leakage?
It removes one specific overlap: the case where a target that looks ahead shares rows across the train/test boundary. It does not fix a leaky feature, a shuffled split, or an event-date join. In my test with a 48-row target, gap=0 scored 0.731 and gap=48 scored 0.730, so it cost nothing and changed nothing.
Sources
[1]pandas, DataFrame.rolling — the API reference forwindow,center,closedandmin_periods, including the default that the window ends on the right edge and the warning that centering looks ahead: pandas.pydata.org/docs/reference/api/pandas.DataFrame.rolling.html[2]pandas, DataFrame.interpolate — the reference for the fill methods used in trap 4, includingmethod,limitandlimit_direction, which decide how far a gap fill reaches: pandas.pydata.org/docs/reference/api/pandas.DataFrame.interpolate.html[3]scikit-learn, TimeSeriesSplit — thegapparameter, defined as the number of samples excluded from the end of each train set before the test set: scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html[4]pandas, merge_asof — the backward-matching join used to attach a value at the time it became available rather than the time it describes: pandas.pydata.org/docs/reference/api/pandas.merge_asof.html[5]scikit-learn, TargetEncoder — the encoder that cross-fits internally so a row's encoding is built without that row's target, withcvandshufflecontrolling the internal splits: scikit-learn.org/stable/modules/generated/sklearn.preprocessing.TargetEncoder.html
