Sam Austinon October 31, 202612 min read

TimeSeriesSplit vs Walk-Forward Validation: Which One Should You Trust?

TimeSeriesSplit vs Walk-Forward Validation: Which One Should You Trust?
Contents

Figure 1: TimeSeriesSplit vs Walk-Forward Validation: Which One Should You Trust? — two names for the same folds, and the four settings that decide whether the score is honest

You have seen both names in tutorials and wondered which one is "the real thing." Plot twist: they are mostly the same thing. The real question is which settings you pick, and whether your validation can see the change that will break your model.

The Short Version

TimeSeriesSplit is scikit-learn's tool [2]. It trains on the past, tests on the next block, then moves forward.

Walk-forward validation is the general idea behind that: train on history, predict the next chunk, step ahead, repeat [4]. Many people write it as a loop by hand.

So are they different? I checked, and on the main point, no.

They Produced Identical Splits

I wrote a plain walk-forward loop: start at a chosen row, test the next 100 rows, then step forward by 100 and repeat. Then I compared it with TimeSeriesSplit(n_splits=8, test_size=100) on a synthetic series of 1,998 rows.

All 8 folds matched: the same training indices and the same test indices. I also tried a sliding window, where old data drops off. The loop with window=300 matched TimeSeriesSplit(max_train_size=300) exactly.

That is the useful takeaway. The label on the method matters less than four settings:

  • n_splits — how many folds.
  • test_size — how many rows each fold predicts.
  • gap — rows skipped between train and test.
  • max_train_size — whether old data expires.

Get those four right and the name on the function stops mattering.

What the Defaults Do

The docs say the default test_size is n_samples // (n_splits + 1), and that each fold trains on everything before its test block [1]. I verified the sizes on my data with 5 splits:

Fold Train rows Test rows
1 333 333
2 666 333
3 999 333
4 1,332 333
5 1,665 333

That matches the docs' formula exactly. Two things follow from the defaults.

The first fold trains on very little data. Its score says more about a small-sample model than about your final one. On 1,998 rows it sees 333 of them, a sixth of the series.

Each test block is big. One fold covers nearly a year if your rows are days, not the one-week block you probably had in mind.

A hand-rolled walk-forward usually uses small test blocks, so you get more folds that mimic "predict the next week, retrain, repeat." You can do the same with TimeSeriesSplit by setting test_size yourself.

Where Your Choice Actually Matters: Change Over Time

Here is the part tutorials skip. I built a series whose rules change at row 1,398. Before the break, y[t] = 0.7*y[t-1] - 0.2*y[t-2] + noise. After it, y[t] = -0.4*y[t-1] + 0.5*y[t-2] + noise. Then I scored a Ridge model on 8 forward folds of 100 rows each, with the two lags as features.

Fold Test rows Expanding window Sliding (300 rows)
0 1196–1295 0.869 0.881
1 1296–1395 0.875 0.870
2 1396–1495 3.191 3.213
3 1496–1595 1.385 0.831
4 1596–1695 1.436 0.971
5 1696–1795 1.352 0.777
6 1796–1895 1.296 0.840
7 1896–1995 0.971 0.853

MAE, lower is better [5]. The blocks start at row 1,196 because two rows were lost to the lag columns. Read the table in three parts.

  1. Before the break: the two are close. 0.872 versus 0.876 averaged over the first two folds.
  2. Right at the break: both fail. Fold 2 spans row 1,396 to 1,495, so it straddles the change, and both encodings of history score above 3.1.
  3. After the break: the gap opens. The expanding window averaged 1.288 over folds 3 to 7, and the sliding window averaged 0.855.

The expanding window keeps training on old rules. The sliding window forgets them. So the right choice depends on your data: if the world changes, a sliding window can adapt, while an expanding window drags the past along.

One nuance worth noticing: the expanding window is not stuck forever. Its fold-3 score of 1.385 improves to 0.971 by fold 7, because by then roughly 500 post-break rows sit in its training set. It recovers, just slowly, and only as fast as new data accumulates.

Does that mean sliding always wins? No. If your relationship stays stable, more history helps, and the expanding window is the better bet. I tested one series with one break.

The Trap That Fools Beginners: Shuffled K-Fold

What if you skip all of this and use plain shuffled K-fold [3]? Here is what I saw with gradient boosting on the same lag features:

Validation scheme MAE
Shuffled 5-fold KFold 0.943
TimeSeriesSplit, 5 folds, average 1.225
Last 300 rows, trained only on earlier rows 1.156

The shuffled number looked about 18% better than the honest hold-out. That gap exists because shuffled folds put rows from the new regime into training. In production, you will not have those.

Notice the second row too. The 5-fold TimeSeriesSplit average came out worse than the single final hold-out, 1.225 against 1.156, because its first fold trains on 336 rows and scores badly. That is not a reason to distrust forward validation. It is a reason to read the per-fold scores instead of only the mean, and to set test_size so the early folds are not starved.

So which one should you trust? The one that matches how you will use the model. If you will predict the future from the past, your validation should do exactly that.

What About Gap?

The gap setting skips rows between the end of training and the start of the test block. The docs describe it as the "number of samples to exclude from the end of each train set before the test set" [1].

You would use it when a target looks ahead. For example, if your target is the average of the next 24 values, the last training rows share information with the first test rows.

I tested that setup with 6 folds of 150 rows, comparing gap=0 with gap=24. Both runs scored 0.269 MAE, and the per-fold numbers matched to two decimals. In this experiment, the gap changed nothing. The target is a 24-row mean, so it is much smoother than the series itself and the absolute errors are small, which leaves little room for a leak to show.

I still think a gap is a sensible precaution for overlapping targets, but I cannot claim my demo proved it matters. Add it when your target windows overlap, and say so in your write-up.

Also remember that gap and test_size count rows, not days. They only mean "24 hours" if your data has one row per hour.

So Which One Do You Use?

Here is my practical rule:

  1. Start with TimeSeriesSplit. It is tested, and it works inside cross_val_score.
  2. Set test_size to match your forecast horizon. Do not accept a default that tests a third of your data at once.
  3. Try max_train_size if you suspect the world changes. Compare it with the expanding default.
  4. Add gap when targets overlap or when features use future-leaning windows.
  5. Write your own walk-forward loop when you need something the class cannot do, like retraining on a schedule or using different data cutoffs per fold. Then check it against TimeSeriesSplit on a small example, as I did.
  6. Keep a final hold-out. Tune with forward folds, then score once on the last block you never touched.

Common Pitfalls

  1. Assuming the two methods differ. They produce the same splits; only the settings differ.
  2. Leaving test_size at the default. You get five huge blocks, and the first fold trains on a sixth of the data.
  3. Reporting only the mean fold score. The per-fold spread is where the regime change shows up.
  4. Using shuffled K-fold on temporal data. It put future rows in training and looked 18% better than the hold-out.
  5. Expecting gap to fix a leaky feature. It only separates train and test targets, and in my test it moved nothing.
  6. Reading gap=24 as 24 hours. It is 24 rows.
  7. Tuning on the final hold-out. Score it once, or it stops being a hold-out.

Wrapping Up

Which one should you trust? Trust the validation that copies your deployment. TimeSeriesSplit and walk-forward validation gave identical splits in my test, so the difference lives in your settings, not the name. Shuffled K-fold, on the other hand, flattered the model by about 18% on a series that changed its rules.

Pick one dataset you care about and run three schemes side by side: shuffled, expanding, and sliding. If shuffled looks much better than the other two, you have found optimism your model will not deliver.

Frequently Asked Questions

Are TimeSeriesSplit and walk-forward validation the same thing?

In effect, yes. TimeSeriesSplit is scikit-learn's implementation of the walk-forward idea: train on the past, test on the next block, step forward, repeat. On my 1,998-row series, a hand-written loop and TimeSeriesSplit with 8 splits of 100 rows produced the same training and test indices in all 8 folds, and a 300-row sliding loop matched max_train_size=300 exactly.

What is the default test_size in TimeSeriesSplit?

n_samples // (n_splits + 1). On 1,998 rows with 5 splits that is 333 rows, so the folds train on 333, 666, 999, 1,332 and 1,665 rows while each test block stays 333 rows. It is the largest test block that still leaves a non-empty first training set when gap is zero.

Should I use an expanding or a sliding window?

Expanding if the relationship is stable, because more history usually helps. Sliding if the world changes, because old rows teach rules that no longer hold. On a series that flipped its coefficients at row 1,398, the sliding window averaged 0.855 MAE after the break against 1.288 for the expanding window, while before the break they were level at 0.872 and 0.876.

Why is shuffled K-fold optimistic for time series?

Because shuffling puts future rows, including rows from later regimes, into training, so the model has effectively seen the answer. On my regime-change series, shuffled 5-fold scored 0.943 MAE against 1.156 on a final hold-out, about 18% better than the model could deliver in production.

What does gap do in TimeSeriesSplit?

It drops that many rows from the end of each training set before the test block starts, so the last training target cannot overlap the first test target. Use it when your target looks ahead, such as a 24-hour average. In my test, gap=0 and gap=24 both scored 0.269 MAE, so the precaution cost nothing and proved nothing either way.

Sources

What are You Looking For?

esc