Sam Austinon November 2, 202615 min read

tsfresh vs Hand-Made Features: Is Automatic Time Series Feature Extraction Worth It?

tsfresh vs Hand-Made Features: Is Automatic Time Series Feature Extraction Worth It?
Contents

Figure 1: tsfresh vs Hand-Made Features: Is Automatic Time Series Feature Extraction Worth It? — one function call produces hundreds of columns, and the question is which ones you can trust

Hand-crafting features takes domain knowledge and afternoons. Automatic extraction takes one function call. So why is not everyone just pressing the button?

I expected to write this post around a stand-in library, because tsfresh would not install. It installed. So everything below is the real package, version 0.21.2, running on Python 3.12 with pandas 3.0.6. That changes the numbers and, in one place, the conclusion.

What tsfresh Says It Does

Per its documentation, tsfresh provides "systematic feature engineering from time-series and other sequential data" and computes characteristics such as maximum, minimum, average and number of peaks [1]. The package ships 77 feature calculators [4], which I counted in the installed module and which the feature list documents one by one.

Then it filters. The filtering page describes the FRESH algorithm in three phases [2]:

  1. Compute features from each series.
  2. Test each feature on its own against the target, producing p-values.
  3. Run the Benjamini-Yekutieli procedure on those p-values to decide what to keep.

The API defaults are fdr_level=0.05, a Mann-Whitney U test for a binary target with a real-valued feature, and Kendall's tau for real-versus-real, though that last parameter is marked "currently unused" in 0.21.2 [3]. The docs describe fdr_level as "the theoretical expected percentage of irrelevant features among all created features" [3].

Limitations it lists itself: it is "not suitable" for streaming data, it does not train models, and it is less suited to "highly irregular time series", because it "uses timestamps only to order observations" [1].

The Experiment

I generated 400 synthetic vibration-style windows of 256 samples and compared three approaches under 5-fold cross-validation with a random forest of 200 trees.

The default EfficientFCParameters turned those 400 windows into 777 columns, 721 of them non-constant. Extraction took 38.5 seconds in a single process, about 96 milliseconds per window. Everything about who wins comes from how I built the scenarios.

One practical note: my first run crashed on Windows with a multiprocessing error about starting a process before the bootstrapping phase. The fix is the if __name__ == "__main__": guard, or n_jobs=1.

Scenario 1: Periodic Impulses

Faulty windows get an impulse every 37 samples, height 2.5 against noise of 1.

Approach Accuracy
Hand-made v1: three FFT bins at the "known" fault rate 0.705
Hand-made v2: autocorrelation at lags 37 and 74 0.850
tsfresh bank, all 721 features 0.930
tsfresh + FRESH selection, inside each fold 0.925

The automatic route won, and my first hand-made attempt was poor. I checked the reason this time: 256 samples divided by 37 is 6.92 cycles, so the impulse train does not land on a whole FFT bin. The fundamental at bin 7 holds about 1.2% of the spectrum's energy, and bins 6 and 8 are roughly 40% weaker than it. Three bins catch a fraction of the signal, which is why 0.705 is the floor here.

The bank had features like maximum, absolute maximum, root mean square and absolute energy, which catch impulses without anyone knowing the interval. That is the real selling point: when you do not know exactly what to look for, a wide net beats a bad guess.

Scenario 2: Context the Window Does Not Contain

Here a fault means the amplitude is 25% too high for the current operating speed, and speed varies from 1x to 4x across windows.

Approach Accuracy
Hand-made: window standard deviation divided by speed 0.992
tsfresh bank, window only 0.628
tsfresh + FRESH selection, window only 0.637
tsfresh + the speed column 0.655

Now the hand-made feature wins big. The bank sees a loud window and cannot tell whether it is loud because of a fault or because the machine is running fast. No statistic computed from the window alone can recover information that is not in the window.

But look at the last row, which is the more interesting result. Your draft had the speed column lifting the bank to 0.775. With 721 features it lifted it to 0.655, because one useful column is diluted by 720 others.

I reran the same scenario with a compact 38-feature subset of the same calculators. The speed column lifted accuracy from 0.632 to 0.820. The context is only usable if the forest can find it.

The Trap in Automatic Selection

Selecting features by their relationship to the label is useful, and dangerous if you do it outside the cross-validation loop. I tested this with pure noise labels, where true accuracy is 0.5, over 20 repeats:

Selection method Mean accuracy
Top-10 chosen on all rows, then cross-validated 0.597 (sd 0.030)
Top-10 chosen inside each fold 0.499
FRESH, fdr_level=0.05 0 features kept in all 20

So the leaky version "found" a ten-point edge from nothing. The FDR rule, which controls the chance of keeping irrelevant features, kept nothing, which is the right answer on noise. If you use any tool like this, put selection inside your pipeline and fit it on training folds only. The tsfresh pages I read give no guidance on this, so it is on you.

Note also how much FRESH keeps when the labels are real: 193 of 721 features in scenario 1, and 134 of 720 in scenario 2. It is a filter, not a feature-count slasher.

Cost

My extraction took 38.5 seconds for 400 windows, and my hand-made statistics took 21 milliseconds. That ratio is roughly 1,800 to 1, and it says less about tsfresh than about the job: 721 features cost far more than 5.

What it does show is the shape of the cost: a wide bank is slower to compute and produces a wider table to hold in memory. Parallelising across cores helps, once you get the multiprocessing guard right.

Do the Timestamps Matter?

The docs say tsfresh uses timestamps only to order observations. I checked: shifting every timestamp by 5,000, and replacing the regular grid with irregular gaps, both changed 0 of 777 features.

That confirms the documented behaviour and also confirms the warning. Interval-agnostic features like number of peaks survive any spacing; features like linear trend assume equal spacing and should be used with care on irregular data [1].

So, Is It Worth It?

My answer, from the real package:

  • Use tsfresh as a baseline, especially early, when you do not know which characteristics matter. It was competitive or better when the signal lived in the window.
  • Use hand-made features when the signal depends on context the series does not carry: speed, load, calendar, other sensors, or a physical relationship. Nothing beat one division in scenario 2.
  • If you combine them, keep the bank narrow. The same speed column was worth 0.19 of accuracy at 38 features and 0.03 at 721.
  • Always select inside cross-validation. The leaky version invented ten points on noise.
  • Check irregular timestamps, since the package treats them as order only.
  • Keep the model honest: score on a time-ordered holdout when the data is a real time series.

Common Pitfalls

  1. Benchmarking a stand-in and calling it tsfresh. I nearly did; the real package changed one conclusion outright.
  2. Selecting features on all rows before cross-validating. That is how a noise dataset scores 0.597.
  3. Assuming the filter keeps a handful of features. FRESH kept 193 of 721 on real labels.
  4. Handing a wide bank one crucial external column. It gets diluted, 0.655 against 0.820 for a narrow bank.
  5. Expecting window statistics to recover context they cannot see. Speed, load and calendar are not in the window.
  6. Forgetting the multiprocessing guard on Windows. The extraction dies before it starts.
  7. Trusting timestamps to carry meaning. Shift them all by 5,000 and not one feature moves.

Best Book to Read

BookDescriptionGet it
Feature Engineering Bookcampby Sinan Ozdemir works through building features by hand for real problems, which is exactly the skill that decides whether an automatic bank helps or just adds columns. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Wrapping Up

Is it worth it? As a first pass, yes: the 721-feature bank beat my best hand-made feature, 0.930 to 0.850, and it got there without knowing the fault period. As a finishing move, no: nothing replaced one line of physics when speed mattered, 0.992 against 0.628.

Run a wide extraction on your data once, look at which features survive selection, and use them as hints for what to hand-build next. Then test both inside the same cross-validation. That one experiment will tell you more than any recommendation, including mine.

Frequently Asked Questions

Is tsfresh worth using instead of hand-made features?

It depends on where the signal lives. In my tests the 721-feature tsfresh bank scored 0.930 against 0.850 for my best hand-made feature when the fault was visible inside the window. When the fault depended on machine speed, the hand-made ratio scored 0.992 and the bank scored 0.628. Use it as a first pass, then hand-build the features it cannot see.

How many features does tsfresh produce?

The package ships 77 feature calculators, and the default EfficientFCParameters settings turned my 400 windows into 777 columns, 721 of them non-constant. The count depends on the calculators you enable, so a compact subset can produce 40 features from the same data.

What does fdr_level mean in tsfresh?

It is the target false discovery rate for the FRESH filter. The documentation describes it as the theoretical expected percentage of irrelevant features among all created features, and the default is 0.05. Lower values keep fewer features, and on pure noise labels the filter kept zero features in all 20 of my repeats.

Why did adding a speed column not help the wide feature bank?

Because one useful column among 721 gets diluted. With the full bank, adding speed lifted accuracy from 0.628 to only 0.655. With a compact 38-feature subset of the same calculators, the same speed column lifted it from 0.632 to 0.820. A wide net makes the external context harder for a tree ensemble to find.

Does tsfresh use the actual timestamp values?

No. Shifting every timestamp by 5,000, or replacing the regular grid with irregular gaps, changed 0 of 777 features. The docs say it uses timestamps only to order observations, and my test confirms that. It also means features such as linear trend assume equal spacing and should be used with care on irregular data.

Sources

What are You Looking For?

esc