Sam Austinon October 31, 202614 min read

7 Data Leakage Traps in Time Series ML (and How to Catch Them)

7 Data Leakage Traps in Time Series ML (and How to Catch Them)
Contents

Figure 1: 7 Data Leakage Traps in Time Series ML (and How to Catch Them) — seven ways a feature quietly reads the answer, and the one cheap test that catches four of them

Your model scores beautifully in validation, then falls apart in production. Congratulations: you have met data leakage, the bug that flatters you right up until it matters.

I built a small synthetic test for each of seven common traps. Some leaked badly. One did nothing at all. I will show both, because a trap you cannot measure is a trap you cannot prioritize.

# Trap Leaky vs honest MAE Detector sees it?
1 Window includes today 0.718 vs 1.110 yes
2 Centered window 0.659 vs 1.110 yes
3 Shuffled split 0.943 vs 3.848 no
4 Interpolation across gaps 0.929 vs 1.068 only near a gap
5 Target encoding on all rows 1.538 vs 4.568 yes
6 Overlapping target, no gap 0.731 vs 0.730 no
7 Event-date join 1.203 vs 2.660 no

MAE, lower is better. The rows use different series, models and splits, so compare each row's two numbers, not the rows against each other.

The Detector You Will Want First

Before the traps, here is a check that catches many of them. It is the perturbation test:

  1. Build your features.
  2. Change the target series from row t0 onward. I added 1,000.
  3. Rebuild the features and compare every row up to and including t0.
  4. If any value moved, that feature peeks at the future.

It is cheap and blunt. One lesson from my own run: a single t0 missed the interpolation leak in trap 4, because that leak only fires when the changed row sits in a gap. Test at many start rows. I used 20.

Feature builder Start rows that flagged it (of 20)
rolling(3).mean() 20
rolling(5, center=True).mean() 20
shift(1).rolling(3).mean() 0
Interpolated lag-1 column 4
Expanding mean over all rows 20

Four of the twenty start rows caught the interpolation leak. Pick one t0 at random and you have a one-in-five chance of noticing. That is why the sweep matters more than the test itself.

The detector cannot see traps 3, 6 or 7. Those involve splits or timing rather than a feature reading y, and I will flag them as we go.

Trap 1: The Window Includes Today

y.rolling(3).mean() includes the current row by default [1]. If you predict y, your feature contains part of the answer.

The catch: the detector flags it, every time. The fix: y.shift(1).rolling(3).mean().

The cost, in my test: a leaky MAE of 0.718 versus 1.110 for the causal version. The leak made the model look about a third better than it is.

Trap 2: The Centered Window

rolling(5, center=True) labels each window at its middle, so it sees two rows ahead [1]. It is fine for smoothing a chart. It is a leak as a model input.

The catch: detector. The cost: 0.659 versus 1.110 for the causal feature, a 41% discount.

Why do people do it? Plots look nicer. Keep centered windows far from your feature code, or name the function so loudly that nobody reuses it.

Trap 3: The Shuffled Split

Plain shuffled K-fold mixes past and future. How much it hurts depends on what your model can memorize.

I gave a gradient-boosted model a numeric time index and a series with a slowly wandering level. The shuffled score was 0.943. The forward score was 3.848. A tree can look up "what happened near row 1,400" when neighbours from both sides sit in training, but it cannot do that for rows beyond the end.

The catch: compare shuffled and forward scores. A gap this size is an alarm. The fix: TimeSeriesSplit, or any forward-only scheme.

Fair warning: this is a deliberately exposed construction. Pure lag features can show a much smaller gap, and in an earlier test of mine a shuffled split barely flattered the model at all.

Trap 4: Filling Gaps With the Future

Interpolation fills a missing value using points on both sides. If your feature is "yesterday's reading" with holes, the point after a hole is today's target.

I deleted 25% of a lag-1 sensor's readings and compared fills:

Fill method MAE
interpolate() 0.929
ffill() 1.068

Interpolation looked 13% better because it borrowed from tomorrow's row [2]. The catch: detector, but only at several start rows. The fix: forward-fill, or fit a fill rule using only past values.

Trap 5: Encodings Built From All the Rows

Target encoding replaces a category with its average target. Compute that average over the whole dataset and each row's own target helps produce its feature.

The size of this leak depends on how many rows each category has:

Levels in the ID Rows per level Leaky MAE Causal MAE
150 13.3 3.066 3.497
1,000 2.0 2.206 4.274
2,000 1.0 1.538 4.568

With 13 rows per level, the leak was worth 12%. With one row per level, it cut the error by 66%. High-cardinality IDs are the danger: the fewer rows behind a category average, the more of that average is the row's own target.

The catch: detector. The fix: an expanding mean of earlier rows only, shift(1).expanding().mean() per group. Scikit-learn's TargetEncoder cross-fits internally for the same reason, so the encoding a row receives never saw that row's target [5].

Trap 6: Overlapping Targets and the Gap

If your target is "the mean of the next 48 values," rows near the train/test boundary share future data. TimeSeriesSplit has a gap parameter for this, which the docs describe as the number of samples excluded from the end of each training set [3].

I tested it with a Ridge model on two lags. The result: MAE 0.731 with no gap versus 0.730 with a gap of 48. No meaningful difference. My earlier test of the same idea also came out flat, 0.269 against 0.269.

So is the trap fake? I would not say that. My two experiments just did not catch it. I would still add a gap when targets overlap, because it costs little, but I cannot tell you it will change your numbers.

Trap 7: Joining by Event Date Instead of Availability

An indicator for Tuesday might be published on Wednesday. Join it by its own date and your model "knows" Tuesday's value on Tuesday.

I built a daily indicator that is published a day late, with a target that depends on the same-day value:

Join key MAE
Event date (leaky) 1.203
Previous day only (available) 2.660

The gap here is by construction, since I made the target depend on the indicator. Real data will vary, but the mechanism is common with revisions, filings and reporting lags.

The catch: the detector cannot see this one, because the indicator does not depend on y. You need timestamps for when each value became known. The fix: join on availability time with pd.merge_asof, which matches backward by default, and add an explicit lag when you lack publish times [4].

A Bonus Non-Finding: The Scaler

Fitting StandardScaler on all rows is a textbook warning. In my run, the fitted mean of one feature was 0.143 on all rows and 0.092 on the first 80%. So the fit does change, by about a third.

I did not score a model with it, so I will not claim how much it matters here. Fit scalers on training data inside a pipeline anyway: it is free.

Your Pre-Flight Checklist

  1. Run the perturbation test on every feature builder, at many start rows.
  2. Use forward-only splits. Treat a big shuffled-versus-forward gap as an alarm.
  3. Shift before you roll.
  4. Fill gaps with past values only.
  5. Build encodings from earlier rows.
  6. Add a gap when targets overlap.
  7. Join on when data became known, not on its label date.
  8. Keep a final hold-out you touch once.

Common Pitfalls

  1. Testing the detector at a single start row. It caught the interpolation leak 4 times out of 20.
  2. Trusting a good score as evidence of no leak. The leak is invisible in the metric by definition.
  3. Assuming every leak shows up as a huge gain. Trap 5 at 13 rows per level was worth 12%, easy to wave off.
  4. Fixing the split and calling it done. A shuffled split was only one of seven traps, and three of them survive a correct split.
  5. Blaming production for a validation bug. If the score drops on deploy, look at the features before the model.
  6. Adding a gap and assuming the target overlap is solved. It moves nothing when the leak is in the feature, as traps 1, 2, 4, 5 and 7 show.

Best Book to Read

CoverBookDescriptionGet it
Cover of “Feature Engineering for Machine Learning” Feature Engineering for Machine Learningby Alice Zheng and Amanda Casari the target-encoding chapter explains, in plain terms, why an aggregate that includes the current row leaks, which is trap 5 here. It also covers the time-based feature traps that make the other six easier to spot in code review. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Wrapping Up

Leakage rarely announces itself. The traps with the biggest measured effects in my tests were the sneaky ones: a window that includes today, an interpolation across a gap, an ID with few rows per level, and a late-published indicator. The one I could not demonstrate, the gap, taught me something too: measure, do not assume.

Pick the model you trust most and run the perturbation test on its feature code today. If a single feature moves, you have just saved yourself a production surprise.

Frequently Asked Questions

What is the fastest way to detect data leakage in time series features?

The perturbation test. Build your features, add a large constant to the target from one row onward, rebuild the features, and compare every row up to that point. If any value moved, the feature read the future. Test at many start rows, because an interpolation leak only fires when the changed row sits next to a gap.

Does pandas rolling include the current row?

Yes. rolling(3).mean() includes the row it is labelled with, because closed=None means the right edge of the window. If the column you are rolling is the target, the feature contains part of the answer. Use y.shift(1).rolling(3).mean() or closed="left" to end the window at the previous row.

Why does interpolation leak in time series?

Because interpolate fills a missing value using points on both sides of the gap. When the missing column is a lag of the target, the fill borrows a future target value. Forward-filling only uses the past. In my test, interpolation scored 0.929 MAE against 1.068 for ffill, and the gain was entirely borrowed from tomorrow.

Is target encoding safe with high-cardinality IDs?

It is most dangerous there. A category with one or two rows gets an encoding that is mostly its own target, so the model can read the answer. In my test, 13 rows per level cut the error by 12%, two rows per level cut it by 48%, and one row per level cut it by 66%. Use an expanding mean of earlier rows, or a cross-fitted encoder.

Does a gap in TimeSeriesSplit stop leakage?

It removes one specific overlap: the case where a target that looks ahead shares rows across the train/test boundary. It does not fix a leaky feature, a shuffled split, or an event-date join. In my test with a 48-row target, gap=0 scored 0.731 and gap=48 scored 0.730, so it cost nothing and changed nothing.

Sources

What are You Looking For?

esc