Outlier Detection and Removal in Python: IQR, Z-Score, and Isolation Forest

Contents

Figure 1: A flagged point is a question, not a verdict — the answer decides whether removing it helps or lies

Three detectors, four tests, and one result that broke the folklore

Some outliers are typos. Some are your best customers. Delete the wrong one and your model gets worse, or your report quietly lies.

Three popular detectors were run on synthetic data where exactly which points were bad was known in advance. Some results were predictable. Isolation Forest's result was not.

The Three Methods in One Minute

• Z-score: flag values more than 3 standard deviations from the mean.
• IQR rule: flag values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR, where IQR is the gap between the 25th and 75th percentiles.
• Isolation Forest: build random trees that split features at random. Per scikit-learn's docs, anomalies get shorter average path lengths because random partitioning isolates them in fewer splits.

A modified z-score using the median and MAD (median absolute deviation) was tested too, since it's a cheap fix for the z-score's biggest weakness. A hand check on [1…9, 100] put the IQR fences at [-3.5, 14.5], flagging only the 100, as expected:

import numpy as np

x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 100])
q1, q3 = np.percentile(x, [25, 75])
iqr = q3 - q1
fences = (q1 - 1.5 * iqr, q3 + 1.5 * iqr)   # (-3.5, 14.5)
flagged = (x < fences[0]) | (x > fences[1])  # only the 100

Test 1: How Many Outliers Can Each Rule Handle?

1,000 normal values (mean 50, sd 10) with outliers from N(120, 5) added, then the outlier count raised. Results, averaged over 20 datasets:

Outlier share	Z > 3 (precision / recall)	IQR 1.5	MAD 3.5
2%	1.00 / 1.00	0.76 / 1.00	0.99 / 1.00
13%	1.00 / 0.00	0.99 / 1.00	1.00 / 1.00
29%	n/a / 0.00	n/a / 0.00	1.00 / 0.81

Here "n/a" means the method flagged nothing, so precision is undefined. Three lessons jump out:

• Z-scores get fooled by the outliers themselves. At 13% contamination, the outliers dragged the mean and inflated the standard deviation so much that none of them looked extreme. Recall hit zero.
• The IQR rule survives longer but breaks too. It handled 13% and failed at 29%, because the outliers started to move the quartiles.
• MAD-based scoring held up best. It still caught 81% at 29% contamination.

Notice the IQR's precision at 2%: 0.76. On pure normal data, the 1.5×IQR rule flags roughly 0.7% of points by design — those false alarms are the price of its sensitivity.

Test 2: Skewed Data Punishes the IQR Rule

Not every extreme value is an error. Clean lognormal data was generated, where nothing is wrong, and each rule's flag share checked (200 datasets):

Rule	Share of clean values flagged
Z > 3	1.9%
IQR 1.5	7.8%
IQR 1.5 on the log scale	0.7%

Then the flagged values were dropped and the mean estimated. The true mean of this distribution is 1.649. Keeping everything gave 1.644. Dropping the IQR-flagged points gave 1.185 — a 28% underestimate, because the legitimate right tail got chopped off.

IMO, that's the quiet danger of "remove outliers." On skewed data, apply the rule on a log scale (or another transformation), or ask whether those big values are real before you delete anything.

Test 3: Where Isolation Forest Humbled Me

Univariate rules only look at one column at a time. So a harder case was built: two features with a 0.95 correlation, plus 20 outliers sitting inside each axis's normal range but breaking the correlation along the opposite diagonal. Results over 10 datasets, 20 true outliers among 1,020 rows:

Detector	Precision	Recall	Avg. flagged
Z > 3 on any column	0.00	0.00	3.6
IQR 1.5 on any column	0.00	0.00	10.3
IsolationForest (contamination known)	0.07	0.07	20.0
IsolationForest (contamination='auto')	0.08	0.70	184.9
LocalOutlierFactor (k=20, known)	0.64	0.64	20.0
EllipticEnvelope (known)	0.68	0.68	20.0

The per-column rules were blind, as expected — each point looks normal on each axis.

The surprise is Isolation Forest. With the contamination rate told to it, it found only 7% of the outliers. With 'auto', it caught 70% but flagged about 185 rows to do it, so most flags were wrong. FYI: this wasn't bad luck — across 20 different random_state values on one dataset, recall ranged from 0.00 to 0.10. It's not a seed problem. The best guess is that axis-aligned random splits struggle with this diagonal structure, though that explanation wasn't tested here.

Don't take this as "Isolation Forest is bad." Scikit-learn's outlier-detection page recommends it for high-dimensional data, and this was a 2-D case built to defeat univariate rules. EllipticEnvelope did well here because the data is Gaussian, which is exactly the assumption the docs say it makes. Pick the detector that fits your data's shape.

A real-world echo of that lesson: an Isolation Forest that cleanly separated injected anomalies in a toy sensor feed flagged almost nothing useful when pointed at a production clickstream dataset — where the interesting rows were unusual not on any single axis but in combinations no single feature made obvious, and the 'auto' contamination default buried them under a flood of ordinary power-users. A detector validated on a toy case has told you nothing about your data's geometry.

Test 4: Does Removing Outliers Help a Model?

A regression with a true slope of 2.0: 200 clean points plus 10 bad y-values shifted by +30. Across 100 datasets:

Method	Mean slope	Spread (sd)
No removal	1.962	0.433
IQR on y	1.979	0.074
Z > 3 on y	1.994	0.072
IsolationForest on (x, y), 'auto'	1.929	0.099
Oracle (drop the true bad 10)	1.994	0.072

The average slope without removal was close to the truth — the problem was the spread: 0.433 versus 0.072 after cleaning. Bad points didn't bias the answer here, but they made any single fit unreliable. Z and IQR matched the oracle because those errors are huge compared with the noise. Isolation Forest did slightly worse, probably because 'auto' also removes some good points, though the count wasn't taken.

So Which One Should You Use?

• One column, roughly bell-shaped, few outliers: z-score or IQR is fine.
• Unsure how many outliers you have: use the MAD-based score. It held up when the z-score collapsed.
• Skewed data: transform first (a log works for lognormal), or you'll delete real values.
• Many features, structure matters: try several detectors, and check what they flag by eye. In this test, LocalOutlierFactor and EllipticEnvelope beat Isolation Forest — but only because of how the data was built.
• Always ask: is this an error, or an interesting case? Investigate before you delete.

One more practical note: with IsolationForest, predict returns −1 for outliers and +1 for inliers, and a lower decision_function score means more abnormal. Mixing those signs up is an easy way to remove exactly the wrong rows. And whatever rule you keep, run it as a checked step in your pipeline — the validation guide covers failing loudly when flag rates shift.

Common Pitfalls

• Deleting flagged rows without looking at them — a flag is a question, not a verdict.
• Running a z-score rule on heavily contaminated data — the outliers hide inside the statistics they corrupted.
• Applying 1.5×IQR to skewed data on the raw scale — you'll trim the real tail and bias every summary after it.
• Trusting a detector's default contamination='auto' — it may flood you with flags or miss most true outliers.
• Assuming a toy-case win generalizes — detector rankings flipped between tests here.
• Confusing the sign conventions — predict −1 means outlier; lower decision_function means more abnormal.
• Treating removal as cleaning — sometimes the outlier is the story (fraud, failures, top customers).

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the best outlier detection method in Python?

It depends on your data's shape and how much you know. For one column, roughly bell-shaped with few outliers, a z-score or IQR rule is fine. If you're unsure how many outliers exist, a MAD-based modified z-score held up best as contamination rose (81% recall at 29% contamination, where the z-score hit zero). For many features with real structure, try several detectors — in these tests LocalOutlierFactor and EllipticEnvelope beat Isolation Forest on a correlated 2-D case. Always inspect flagged rows before deleting anything.

When should you not remove outliers?

When the extreme values might be real. On clean lognormal data, the 1.5×IQR rule flagged 7.8% of perfectly valid points, and deleting them cut the estimated mean from the true 1.649 to 1.185 — a 28% underestimate from chopping off the legitimate right tail. Not every extreme value is an error; some are your best customers. On skewed data, transform first (a log works for lognormal), apply the rule on the transformed scale, and investigate flagged rows before you delete anything.

Why does the z-score fail when there are many outliers?

Because the outliers corrupt the statistics the z-score depends on. In these tests, at 13% contamination the outlier group dragged the mean and inflated the standard deviation so much that none of them looked more than 3 standard deviations away — recall hit zero and the method flagged nothing. The MAD-based alternative (median and median absolute deviation) resists this because the median ignores extremes: it still caught 81% of outliers at 29% contamination.

Does Isolation Forest work well?

It depends on the data. Scikit-learn recommends it for high-dimensional data, but in a 2-D case built to defeat univariate rules — two features correlated at 0.95 with outliers lying along the opposite diagonal — it found only 7% of outliers when told the contamination rate, and with contamination='auto' it caught 70% while flagging about 185 of 1,020 rows, so most flags were wrong. Random axis-aligned splits struggle with diagonal structure. Pick the detector that fits your data's shape, and remember predict returns −1 for outliers while a lower decision_function score means more abnormal.

Wrapping This Up

The z-score fails when outliers inflate the standard deviation. The IQR rule is sturdier but over-flags skewed data and still breaks with enough contamination. Isolation Forest isn't a magic upgrade — it struggled on a correlated 2-D case where other methods succeeded.

Take one column of your own data, plot it, and run two detectors side by side. Look at the flagged rows before you remove any.

Related Articles

• How to Handle Missing Data in Python: Imputation Methods Compared
• Feature Scaling Explained: StandardScaler vs MinMaxScaler vs RobustScaler
• Data Preprocessing for Machine Learning: Complete Step-by-Step Guide
• Data Validation with Great Expectations: Catch Bad Data Early