Sam Austinon October 31, 202615 min read

Rolling Window Features in Pandas: Moving Averages for ML

Rolling Window Features in Pandas: Moving Averages for ML
Contents

Figure 1: Rolling Window Features in Pandas: Moving Averages for ML — a rolling window hands the model context, and one missing shift hands it the answer

Your model wants context, and a single row gives it none. Rolling windows hand it the last few days of history as a handful of numbers. They also make it easy to cheat without noticing, so we will cover both.

What a Rolling Feature Actually Is

A rolling window slides along your series. At each row, pandas summarizes the last N values into one number: a mean, a standard deviation, a slope. Each summary becomes a column your model can use.

Three behaviours decide whether those columns are any good:

  • Integer windows count rows. rolling(7) means seven rows, whatever dates they carry.
  • Offset windows count time. rolling("7D") means seven calendar days, however many rows fall inside.
  • The window includes the current row by default. The docs say closed=None means "right", so the last point is part of the calculation [2].

That last one causes most of the trouble. We will get there.

The Basic Toolkit

Start with the usual suspects, computed on the previous values (explained below):

r = y.shift(1).rolling(7)
f["mean_7"] = r.mean()
f["std_7"]  = r.std()
f["min_7"]  = r.min()
f["max_7"]  = r.max()
  • Mean tells the model where the series has been lately.
  • Std tells it how jumpy things are. Volatility often matters more than level.
  • Min and max give it the recent range, which helps with spikes and floors.

One detail bites people who mix libraries: pandas' rolling std defaults to ddof=1, while NumPy's np.std defaults to ddof=0. In my demo the pandas result matched np.std(ddof=1) on every row and did not match plain np.std. Mix the two and your features will differ by a small, annoying factor.

Trend Features: Slope Without the Slowdown

Want the trend over the last seven days? The obvious way is apply with a fitted slope:

def slope(a):
    return np.polyfit(np.arange(len(a)), a, 1)[0]

f["slope_7"] = y.shift(1).rolling(7).apply(slope, raw=True)

It works, but it is slow. The faster route uses the fact that a regression slope equals the covariance of x and y divided by the variance of x [4]. Pandas has rolling versions of both:

x = pd.Series(np.arange(len(y), dtype=float), index=y.index)
f["slope_7"] = y.shift(1).rolling(7).cov(x) / x.rolling(7).var()

On my synthetic data, the two versions matched on every row, with a maximum difference of 2.7e-15. How much does the shortcut matter?

I timed one run each on 100,000 rows with a window of 30:

Method Time
apply(..., raw=False) 12,964 ms
apply(np.mean, raw=True) 391 ms
built-in .mean() 5.1 ms

These are this machine's numbers, so treat the ratios as the point rather than the milliseconds. raw=True passes NumPy arrays instead of Series, and it was 33 times faster than the Series version [1]. The built-in still beat raw=True by about 77 times. Reach for apply only when no built-in does the job.

Exponential Weighting: Moving Averages With a Memory

A plain rolling mean treats every value in the window equally, then forgets the 8th-oldest completely. Is that how you would want recent history to work? Probably not.

Exponentially weighted features fix this. Recent rows count more, and old rows fade out:

f["ewm_7"] = y.shift(1).ewm(span=7, adjust=False).mean()

The docs define span through alpha = 2 / (span + 1) [3]. With adjust=False, pandas computes the average recursively: y_t = (1 - alpha) * y_(t-1) + alpha * x_t. I coded that recursion by hand and it matched pandas on every row.

Two honest comparisons with the rolling mean:

  • EWM never forgets completely. Old values shrink toward zero weight but never reach it. A rolling window drops them at a hard edge.
  • EWM needs no warm-up by default. min_periods defaults to 0 for ewm, so the early rows have values. A rolling(7) leaves the first six empty. Those early EWM values rest on very few observations, so do not over-trust them.

The NaN Problem: min_periods

By default, an integer window demands a full set of observations. One missing value contaminates several rows.

I tested it on [1, 2, NaN, 4, 5, 6] with a window of 3:

Setting Result
Default [NaN, NaN, NaN, NaN, NaN, 5.0]
min_periods=2 [NaN, 1.5, 1.5, 3.0, 4.5, 5.0]

Under the default, five of six rows came back NaN. Two come from the warm-up, and three come from windows containing the NaN. With min_periods=2, only the first row stayed NaN.

So should you set min_periods low everywhere? No. A mean from two points and a mean from thirty are different quality features, and your model cannot tell them apart. If you loosen it, add a count column (rolling(7).count()) so the model knows how much evidence sat behind each value.

Offset Windows and Missing Days

Real data has gaps. Weekends vanish, sensors die, someone forgets to run the job. What does rolling(7) do then?

I deleted ten days from a daily series and checked a row two positions after the hole:

  • rolling(7) reached back across 16 calendar days to find seven rows. Your "weekly average" quietly became a two-week average.
  • rolling("7D") used only the 3 rows that actually fell inside the last seven days.

Neither is automatically right. If "the last seven observations" is your concept, use the integer window. If "the last week" is your concept, use the offset window.

One catch: offset windows default to min_periods=1 [2]. That means the first row gets a "7-day mean" computed from exactly one observation. In my run, the first value came from a single row. Set min_periods explicitly when you use offsets.

The Leak: Why shift(1) Comes First

Here is the part that ruins model evaluations. You want to predict y at time t. The default rolling mean at time t includes y[t]. So you have handed the model a slice of the answer.

The fix is one line:

leaky  = y.rolling(7).mean()              # includes today's value
causal = y.shift(1).rolling(7).mean()     # ends yesterday

Does the leak always wreck your score? Not equally. I tested the same model and the same time-ordered cross-validation (five TimeSeriesSplit folds), changing only the window size [5]. The series has deliberately heavy noise, so history predicts little and the leak shows clearly:

Window Shifted MAE Unshifted MAE Unshifted looks
3 16.31 10.55 1.55x better
7 15.68 12.54 1.25x better
28 15.95 13.28 1.20x better

The pattern makes sense. Each row carries a weight of 1/window in the mean, so short windows leak more. But look at the 28-day row. The leak shrank, and it still made the model look 20% better than it is. A long window hides the leak and does not remove it.

One caveat in the other direction. On an earlier attempt with a different series, strong lag features and a 24-step window, the leak barely moved the score. Leakage size depends on your data, so you cannot check for it by eyeballing a metric. Fix it by construction instead.

Two Ways to Stay Causal

Option 1: shift, then roll.

y.shift(1).rolling(7).mean()

Option 2: close the window on the left.

y.rolling(7, closed="left").mean()

In my test, both versions produced the same values wherever both existed, with the same seven leading NaNs. Pick whichever reads better to your team. I prefer shift(1) because it is hard to misread and works the same with ewm.

Also avoid center=True for features. The docs say it labels the window at its centre, which means it looks ahead [1]. It is fine for plotting a smooth curve. It is wrong for a model input.

A Quick Checklist Before You Train

  1. Shift first. Every rolling feature should end at t − 1.
  2. Choose rows or time deliberately (7 versus "7D").
  3. Set min_periods on purpose, and add a count column if you loosen it.
  4. Prefer built-ins over apply. When you must use apply, pass raw=True.
  5. Split by time, never randomly. A shuffled split puts future rows in training.
  6. Test for leaks. Change y from some row onward and confirm no feature in earlier rows moves.

Wrapping Up

Rolling features are cheap, fast and effective, and they come with one sharp edge: the window includes the present by default. Shift before you roll, choose between row and time windows on purpose, and treat min_periods as a real decision.

Pick one feature you already use, say a 7-day mean, and rewrite it with shift(1). Then rerun your cross-validation. If the score drops, you just found a leak. Better to find it now than in production.

Frequently Asked Questions

Why does shift(1) come before rolling in pandas?

Because rolling includes the current row by default, so a 7-day mean at time t contains y[t], the value you are trying to predict. Calling shift(1) first moves every value down one row, so the window ends at yesterday. It is the cheapest way to make a rolling feature causal.

What is the difference between rolling(7) and rolling("7D")?

rolling(7) counts seven rows, whatever dates they carry, so a gap makes it reach further back in calendar time. rolling("7D") counts seven calendar days and takes however many rows fall inside. Use the integer window for the last seven observations, and the offset window for the last week.

What does min_periods do in pandas rolling?

It sets how many non-missing observations a window needs before it returns a value. The default for an integer window is the window size, so one missing value turns several rows into NaN. Lowering it fills more rows, but a mean of two points is not the same quality as a mean of thirty, so add a rolling count column if you loosen it.

Why is pandas rolling std different from numpy std?

Pandas uses ddof=1 by default, which divides by n−1, while NumPy uses ddof=0, which divides by n. On the same window the two differ by a small factor. Pass ddof=0 to pandas, or ddof=1 to NumPy, if you need them to match.

Is rolling(7, closed="left") the same as shift(1).rolling(7)?

Yes for the values they produce. Both exclude the current row and leave the same number of leading NaNs. shift(1) also works with ewm and reads more explicitly, while closed="left" keeps the window length in one place, so pick whichever your team reads faster.

Sources

What are You Looking For?

esc