Contents
Figure 1: Cleaning Sensor Data: Z-Score and Rolling Rules for Time Series Outliers — the readings come from a sensor, and the sensor does not tell you when it is lying
Sensor data lies in creative ways. It spikes, freezes, and sometimes just changes level for no reason you were told about. A single z-score rule cannot handle all of that, so I tested four on a series where I knew exactly what I had broken.
The Test Series
I built 10,000 synthetic sensor readings: a drift of 8 units across the run, a repeating cycle with period 288 and amplitude 3, and Gaussian noise with a standard deviation of 0.5. Then I injected:
- 40 isolated spikes of 5 to 10 units, up or down.
- 6 bursts of 5 consecutive spikes, 30 points in total.
- 3 flatlines of 40 repeated values, a stuck sensor.
- One level shift of +10 at point 6,000.
The 70 spike and burst points are the labelled outliers. The flatlines and the level shift are scored separately, because they are different kinds of problem. Every number below comes from one run of this one series, with a fixed seed.
Rule 1: The Global Z-Score
The classic: subtract the overall mean, divide by the overall standard deviation, and flag anything beyond a threshold.
Result: it found 0 of 70. At threshold 4 nothing crossed, and at threshold 3 still nothing crossed. The largest global z-score in the entire series was 2.49.
Why? The drift, the cycle and the level shift all inflate the overall standard deviation, which came out at 7.3. A 10-unit spike divided by 7.3 is 1.4. A global z-score only works when "normal" stays put, and sensor data almost never does.
Rule 2: The Rolling Z-Score
Compute the mean and standard deviation over a window of recent points, then compare each new point against them. I used a trailing window of 25 points that excludes the current value, so the rule works in real time.
Result at threshold 4: recall 0.71, precision 0.71. It caught 38 of the 40 isolated spikes, 95%, but only 12 of the 30 burst points, 40%.
That gap is the whole story, and this time I did isolate the mechanism. The mean absolute z-score by position inside a burst was:
| Position in burst | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Mean absolute z | 11.6 | 5.5 | 3.4 | 2.5 | 2.1 |
Only the first two points of a five-point burst clear a threshold of 4. Earlier points in the burst enter the window, inflating both its mean and its standard deviation, so later points look less extreme. This is called masking, and the table is what it looks like when you measure it.
My first run had a different bug: I compared z > 4 instead of |z| > 4, so I caught only the upward spikes. The downward ones scored -12 and sailed through. Check the sign before you trust a recall number.
Rule 3: The Hampel Filter
The Hampel filter swaps the mean and standard deviation for the median and the median absolute deviation (MAD), which are much harder to inflate [1]. The usual form flags a point when it lies outside the moving median plus or minus a threshold times 1.4826 times the MAD [2]. That 1.4826 constant makes the MAD comparable to a standard deviation under Gaussian data, and in SciPy scale="normal" gives exactly that conversion [3].
My version uses a trailing window of 25 and does not replace values.
Result at threshold 4: recall 1.00, including every burst point. The mean absolute Hampel score by position inside a burst was 11.0, 12.1, 11.4, 10.8 and 9.7, flat where the rolling z collapsed.
The price is precision of 0.50: 70 false flags, 57 of them in steady state and 13 just after the level shift. Six of the 57 sit within a dozen rows of a flatline ending, because the sensor comes back to life and the jump out of the stuck value looks like a spike.
The Tuning Trade-Off
I ran a sweep over window size and threshold. False flags below count steady-state clean points, excluding the level-shift stretch:
| Rule | Window | K=3 recall / false flags | K=4 recall / false flags |
|---|---|---|---|
| Rolling z | 15 | 0.73 / 223 | 0.64 / 36 |
| Rolling z | 25 | 0.84 / 143 | 0.71 / 18 |
| Rolling z | 49 | 0.93 / 77 | 0.81 / 12 |
| Hampel | 15 | 1.00 / 438 | 1.00 / 132 |
| Hampel | 25 | 1.00 / 281 | 1.00 / 57 |
| Hampel | 49 | 1.00 / 123 | 0.99 / 20 |
Three honest takeaways.
- Threshold 4 beat threshold 3 on false flags in all twelve settings, and recall never improved with the higher threshold: it fell in four settings and stayed flat in two.
- In my sweep the longer window won on both columns, which is the opposite of the usual story. With a 288-point cycle, a 49-point window still tracks the cycle, and averaging over more points gives a quieter scale. The real trade-off here was between the rules, not between windows: Hampel bought 0.99 recall at 20 false flags, while rolling z got 0.81 at 12.
- A MAD estimated from 15 points is noisy, which is why short-window Hampel false-flagged 438 times at threshold 3. The trailing window also lags the cycle, so it struggles on the steep stretches.
I also tried a version that fits a line to the window first. At a 49-point window it reached recall 0.89 with 88 false flags at threshold 3, and 0.83 with 13 at threshold 4. A more complicated rule did not remove the trade-off.
I chose window 25 and threshold 4 for the main table after looking at this sweep on the same data. That is in-sample tuning, so treat my headline numbers as optimistic. Tune yours on labelled data you did not use to pick the settings.
Special Cases
The level shift. After the +10 jump, the trailing rules flagged a few points until the window filled with the new level: 2 for rolling z and 13 for Hampel in the next 200 points. Then they went quiet. None of them reports "the process changed" as a separate event, so a shift needs its own detector, such as a change-point search [5].
The flatline. A stuck sensor repeats one value. Inside my three flatlines the trailing MAD was exactly 0 at 81 points, 27 per flatline, because a fully flat window has no spread to compare against. Hampel flagged none of the 120 flatline points, and neither did rolling z. A deviation of zero never beats a threshold times zero.
A run-length rule, ten identical values in a row, found all 3 flatlines and produced no flags on clean data. That is the rule you want for a stuck sensor, and it is ten lines of code.
Causal or Centered?
A centred window uses points on both sides, which is fine for cleaning a stored file, but it peeks at the future. I added 1,000 to every value from row 4,000 onward and recomputed the scores of earlier rows. The 12 rows within half a window changed for the centred version, rows 3,988 to 3,999, and none changed for the trailing one.
Centred z also scored best on precision, 1.00, but recall was only 0.37, and it caught 0 of the 30 burst points. A centred window sees the whole burst at once, so the burst inflates its own reference scale even harder. If the result feeds a live alert or a model feature, use trailing windows.
After Flagging: Fill Carefully
I replaced the 70 correctly flagged outliers and compared each fill with the true underlying signal:
| Fill | Mean absolute error |
|---|---|
| Left alone | 7.56 |
| Forward-fill | 0.64 |
| Interpolate | 0.54 |
Both fills beat leaving the spikes in, by more than a factor of ten. Interpolation was slightly closer, but it uses the next good point. When I changed later values, the interpolated fill at row 5,670 changed and the forward-fill did not. For real-time work, forward-fill is the safe default [4].
On bursts alone the gap narrows: 0.63 for forward-fill against 0.59 for interpolation. Both work because the underlying signal moves slowly, which will not be true for every sensor.
A Practical Recipe
- Start with physics: remove impossible values, negative temperatures or readings beyond the sensor's range, before any statistics.
- Use a trailing Hampel rule for spikes, with the window short enough to follow your cycle.
- Raise the threshold until false flags are tolerable, then check recall on labelled examples.
- Add a run-length rule for stuck sensors.
- Handle level shifts separately, with a change-point search or a rule on rolling medians.
- Fill with the past only if the pipeline is live.
- Keep the flags as a column. A flag can itself be a useful feature.
Common Pitfalls
- Trusting a global z-score on a series with drift or a level shift. It found 0 of 70 here.
- Forgetting the absolute value. My first run caught only upward spikes, because a downward spike has a negative z.
- Assuming a burst behaves like one spike. The rolling window masks it, 11.6 down to 2.1 across five points.
- Estimating a MAD from 15 points. It is noisy enough to false-flag 438 clean points at threshold 3.
- Treating a flatline as clean data. The MAD is zero, so the rule cannot fire, and 120 stuck points pass unlabelled.
- Using a centred window in a live pipeline. It changed 12 earlier scores when I edited the future.
- Interpolating when the pipeline is real-time. It reads ahead, and the fill changes when later data arrives.
- Tuning the threshold on the same data you report. I did that, and my headline numbers are optimistic because of it.
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Practical Statistics for Data Scientists | covers z-scores, the median absolute deviation and robust summaries with runnable R and Python, which is the groundwork behind every rule measured here. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Wrapping Up
The fancy part is not the formula. It is picking a rule that survives your data's shape: a global z-score failed completely, a mean-based rule got fooled by bursts, and the robust rule caught everything while crying wolf. A flatline slipped past all of them.
Pull one sensor series, hand-label ten anomalies, and run a trailing Hampel filter at two thresholds. If you cannot hit your recall target at a tolerable false-flag rate, your next fix is the window, not the threshold.
Frequently Asked Questions
Why does a global z-score miss spikes in sensor data?
Because the overall mean and standard deviation are computed over the whole series, and drift, a seasonal cycle and any level shift all inflate them. In my 10,000-point test the overall standard deviation was 7.3, so a 5 to 10 unit spike never reached a threshold of 4. The largest global z-score in the entire series was 2.49, and the rule found 0 of 70 labelled outliers.
Why does a rolling z-score miss spikes inside a burst?
This is masking. The first spike in a burst enters the trailing window, inflating both its mean and its standard deviation, so the following spikes look less extreme. In my test the mean absolute z-score by position inside a burst fell from 11.6 to 5.5, 3.4, 2.5 and 2.1, so only the first two points of a five-point burst cleared a threshold of 4.
What window and threshold should a Hampel filter use?
Start with a window of 25 and a threshold of 4, then check recall on hand-labelled anomalies. In my sweep a threshold of 4 reduced false flags in all twelve settings I tried and never improved recall. The Hampel rule caught every spike at window 25, but it also produced 70 false flags, so the window and the threshold are a compromise, not a setting you can optimize on one metric.
Why did the Hampel filter not catch the stuck sensor?
A flatline has no spread, so the median absolute deviation of the window is exactly zero. The rule compares a deviation against that scale, and a deviation of zero never exceeds a threshold times zero. In my test the trailing MAD was exactly 0 at 81 points and the filter flagged none of the 120 flatline points. A run-length rule, ten identical values in a row, found all three flatlines and produced no flags on clean data.
Should I forward-fill or interpolate after removing outliers?
Forward-fill if the pipeline is live, interpolation if you are cleaning a stored file. Interpolation was slightly closer to the truth in my test, 0.54 average error against 0.64, but it reads the next good point. When I changed later values, the interpolated fill at row 5,670 changed and the forward-fill did not.
Sources
[1]Wikipedia, Hampel test — the window of neighbouring points, the median and MAD as robust scale, and replacement of an outlier by the window median: en.wikipedia.org/wiki/Hampel_test[2]Rick Wicklin, The Hampel identifier: Robust outlier detection in a time series (The DO Loop, SAS) — the rule that observations outside median ± h·(1.4826·MAD) are outliers, the h=3 default, and why the 1.4826 constant makes MAD estimate a standard deviation: blogs.sas.com/content/iml/2021/06/01/hampel-filter-robust-outliers.html[3]SciPy, median_abs_deviation —scale="normal"divides by the inverse normal quantile at 0.75, about 0.67449, which is the 1.4826 conversion in reverse: docs.scipy.org/doc/scipy/reference/generated/scipy.stats.median_abs_deviation.html[4]pandas, DataFrame.interpolate —method="linear"is the default and treats values as equally spaced, which is the interpolation used in the fill comparison: pandas.pydata.org/docs/reference/api/pandas.DataFrame.interpolate.html[5]ruptures, change point detection in Python — the library I would reach for when a level shift needs its own detector rather than a threshold on residuals: centre-borelli.github.io/ruptures-docs
