Feature Selection Methods: Filter, Wrapper, and Embedded Explained

Contents

Figure 1: Fewer columns, clearer signal — selection is what keeps 100 features from becoming 100 ways to fool yourself

Three families of methods, one noise dataset, and a score that lied

More features sound like more information. Often they're more noise, more training time, and more ways to fool yourself. Feature selection is the fix, but there are three families of methods, and they fail in different ways.

All three were run on synthetic data where exactly which features mattered was known in advance. The winners weren't who was expected, and one mistake turned pure noise into 83% accuracy.

The Three Families in One Minute

• Filter methods score each feature on its own, before any model. Examples: SelectKBest with an F-test or mutual information, and VarianceThreshold. They're fast.
• Wrapper methods train a model repeatedly on different feature subsets. Examples: RFE (recursive feature elimination) and SequentialFeatureSelector. They're slower.
• Embedded methods select features while a model trains. Examples: L1-penalized logistic regression and tree-based importances, both used through SelectFromModel.

Scikit-learn's docs describe the cost difference plainly: sequential selection fits many models, RFE needs one fit per step, and SelectFromModel needs a single fit.

The Experiment

200 training rows with 100 features, of which only 5 were informative. Each method picked 5 features. Then: how many true features did it find, and how well did a logistic regression perform on 2,000 fresh test rows? Everything averages 20 datasets.

Method	True found (of 5)	Test accuracy	Seconds per dataset
Filter: F-test	3.30	0.876	0.00
Filter: mutual information	2.95	0.874	0.24
Wrapper: RFE	3.90	0.898	0.07
Wrapper: forward SFS	2.65	0.878	5.95
Embedded: L1 logistic	3.25	0.892	0.01
Embedded: random forest importance	4.05	0.895	0.36
Reference: the 5 true features	5.00	0.907	—
Reference: all 100 features	—	0.807	—

Three things stand out:

• Any selection beat using all 100 features. Accuracy rose from 0.807 to roughly 0.87–0.90. With 200 rows and 100 features, the extra columns mostly gave the model room to overfit.
• No method found all 5. The best, random forest importance, averaged 4.05.
• The expensive method lost. Forward selection took about 6 seconds per dataset, versus a hundredth of a second for the F-test, and it found the fewest true features (2.65). IMO, don't reach for the slowest method first — it isn't automatically the smartest one.

One caveat: the data was built so each class has two clusters, which makes it hard for a linear model. Even the 5 true features only reached 0.907. Your data will behave differently.

Where Filters Fail: Interactions

Filter methods score features one at a time. That's their blind spot. A classic trap was tested: the label depends on the product of two features (an XOR-style pattern), with 20 noise features alongside. No single feature carries any signal by itself. Did each method pick the two real ones? (Average found among the top 2, over 20 datasets.)

Method	True features found (of 2)
F-test	0.15
Mutual information	0.15
L1 logistic	0.15
Random forest importance	2.00

The univariate filters and the linear L1 model were blind, since both features look useless alone. The random forest found both every time, because trees can split on one feature and then the other. Match the method to the structure you expect: if you suspect interactions, a linear or univariate selector can discard exactly what you need.

The Mistake That Fakes a Great Model

This is the one to remember. Pure noise: random features, random labels, 100 rows, 1,000 columns. The true accuracy is 50%, a coin flip. Then feature selection ran two ways:

• Select the 20 best features on all the data, then cross-validate: accuracy 0.832.
• Put the selector inside a pipeline, so it refits within each training fold: accuracy 0.504.

Same data, same model, and one version "discovered" an 83% classifier in random noise. The leaky version let the selector see every label, including the ones in the test folds, so it picked features that happened to match them. The honest version scored 50%, which is the right answer.

FYI: scikit-learn's guide recommends putting selection inside a Pipeline as the preprocessing step. The guide doesn't spell out the leakage reason — that reasoning is backed by the experiment above:

from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("select", SelectKBest(f_classif, k=20)),
    ("model", LogisticRegression(max_iter=1000)),
])

# the selector refits on each training fold — no leaked labels
pipe.fit(X_train, y_train)

A real-world echo: a churn model once showed a flattering 0.94 AUC after SelectKBest was run on the full dataset — and dropped to the low 0.6s once the selector moved inside the pipeline and the same cross-validation ran honestly. The features hadn't changed; only where the selector was allowed to look. If a result looks too good, move your preprocessing inside the pipeline first and watch what survives.

Does Selection Help on Real Data?

On the breast cancer dataset (30 features), the selector went inside the pipeline with 5-fold cross-validation repeated 5 times:

Features kept	F-test	RFE
3	0.9413	0.9462
5	0.9473	0.9655
10	0.9543	0.9670
20	0.9789	0.9758
All 30	0.9772	—

Here, selection didn't help. Using all 30 features scored 0.9772, and cutting to 20 gave about the same; cutting further lost accuracy. With 569 rows and 30 features, the model had no overfitting problem to fix. Selection still has other benefits — faster training, a simpler model to explain — but don't expect an accuracy boost every time.

Which Family Should You Use?

• Start with a cheap filter: VarianceThreshold removes constant columns for free. Then try SelectKBest as a fast baseline.
• Many features, few rows: selection helped most here (0.807 to about 0.89). Try RFE or an embedded method.
• Suspect interactions: use a tree-based embedded method. The XOR test shows why.
• Have time to spare: wrappers like forward selection are thorough but slow, and they weren't better in this test.
• Always: keep the selector inside your pipeline, and cross-validate the whole thing — the same rule the preprocessing guide applies to scaling and imputation.

A few practical details from the docs: chi2 only works on non-negative features, and using a regression scoring function on a classification problem gives "useless results." For L1 models, a smaller C keeps fewer features. And when a correlated block of features makes selection unstable, PCA is the projection-based alternative to picking winners.

Common Pitfalls

• Selecting features before cross-validation — the 0.832-on-noise trap.
• Assuming more features help a small dataset — 100 columns of mostly noise scored 0.807 against 0.89 with 5.
• Using a univariate filter when interactions matter — it scores both XOR features at zero and drops them.
• Paying the wrapper's runtime by reflex — forward selection was 600× slower than the F-test and found fewer true features.
• Expecting selection to boost accuracy on an easy dataset — on breast cancer it changed nothing.
• Forgetting chi2's non-negativity requirement — or mixing a regression scorer into a classification task.
• Ignoring the selector's stability — with 200 rows, even the best method found 4.05 of 5 true features.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the difference between filter, wrapper, and embedded feature selection?

Filter methods score each feature on its own before any model runs — SelectKBest with an F-test or mutual information, VarianceThreshold — so they're fast. Wrapper methods train a model repeatedly on different feature subsets — RFE and SequentialFeatureSelector — so they're slower. Embedded methods select features while a model trains, such as L1-penalized logistic regression or tree-based importances used through SelectFromModel. Per scikit-learn's docs, sequential selection fits many models, RFE needs one fit per step, and SelectFromModel needs a single fit.

Does feature selection improve model accuracy?

Sometimes, and not where people expect. In these tests, with 200 rows and 100 features (only 5 informative), any selection method lifted accuracy from 0.807 to roughly 0.87–0.90 — the extra columns mostly gave a small dataset room to overfit. But on the breast cancer dataset (569 rows, 30 features), selection didn't help at all: using all 30 features scored 0.9772 and cutting to 20 gave about the same. Selection reliably buys smaller, faster, simpler models; accuracy gains depend on whether you had an overfitting problem to fix.

Why should feature selection be inside a pipeline?

Because a selector fit on all the data leaks test-fold labels into training. In the noise-only test — random features, random labels, true accuracy 50% — selecting the 20 best features before cross-validation produced 0.832 accuracy, while the same selector refit inside each training fold scored 0.504, the honest answer. The leaky selector picked features that happened to match the held-out labels. scikit-learn's guide recommends putting selection inside a Pipeline; the leakage reason is the experiment behind it.

Which feature selection method should I use first?

Start cheap: VarianceThreshold removes constant columns for free, then SelectKBest as a fast baseline. If you have many features and few rows, try RFE or an embedded method — they helped most in these tests (0.807 to about 0.89). If you suspect interactions between features, use a tree-based embedded selector: in the XOR test only random forest importance found both true features, while F-test, mutual information, and L1 logistic found 0.15 of 2. Whatever you choose, keep the selector inside the pipeline and cross-validate the whole thing.

Wrapping This Up

Filters are fast but blind to interactions. Wrappers are thorough but slow, and they didn't win. Embedded methods offered the best balance in these tests, with random forest importance finding the most true features. On the breast cancer data, selection didn't improve accuracy. Whatever you choose, selecting before cross-validation can fake a great score from noise.

Take one dataset of your own, wrap a selector and model in a pipeline, and compare its cross-validated score against using every feature.

Related Articles

• Encoding Categorical Variables: One-Hot, Label, and Target Encoding
• Data Preprocessing for Machine Learning: Complete Step-by-Step Guide
• PCA Tutorial: Dimensionality Reduction in Python
• Building a Feature Engineering Pipeline with Scikit-learn Pipelines