Data Leakage in Machine Learning: How to Spot and Prevent It

Contents

Figure 1: Leakage is a hole you can't see from inside the notebook — the score looks fine right up until it doesn't

Five leaks, measured on data where the truth was known

Your model scores 95% AUC. You celebrate, ship it, and it performs like a coin flip. Nine times out of ten, the culprit isn't the model — it's leakage.

Leakage means your model saw information during training that it won't have at prediction time. Scikit-learn's guide defines it exactly that way, and warns it produces "overly optimistic performance estimates." Five leaky setups were built on synthetic data where the truth was known in advance, measured how much each one inflated the score, and a small audit function was written to catch the worst offender.

What Leakage Is, In One Sentence

Any information that reaches the model during training but won't exist at prediction time is leakage — and it doesn't look like a bug. It looks like a great score. The five classic forms:

1. Preprocessing fitted before the split.
2. A column that only exists after the outcome.
3. Time shuffled across train and test.
4. The same entity sitting on both sides of the split.
5. Duplicates leaking through.

Each one below is measured.

Leak 1: Preprocessing Before the Split

The most common leak is also the sneakiest: you select features (or scale, or impute) on the full dataset, then cross-validate.

The experiment: pure noise — 200 rows, 10,000 random features, random labels. True accuracy is 50%. Then pick the 25 "best" features two ways, averaged over 20 repeats:

Approach	Accuracy
Select on all rows, then cross-validate	0.837
Selector inside the pipeline	0.501

An 84% classifier built from random numbers. The selector saw every label, including the test folds, so it chose features that happened to line up with them by chance. Scikit-learn's guide shows the same experiment and reports 0.76 for the leaky version and about 0.5 for the right one. FYI: the guide says the risk applies to almost every transformer, including StandardScaler, SimpleImputer, and PCA — the same trap the preprocessing guide warns about at every step.

The fix is mechanical: put every fitted step in a Pipeline.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif

# wrong: selector.fit(X_all, y_all) then cross_validate(...)
# right: fit only ever sees the training fold
pipe = make_pipeline(StandardScaler(), SelectKBest(f_classif, k=25),
                     LogisticRegression(max_iter=1000))

The guide's rule is blunt: "never call fit on the test data." And it's the same discipline as feature selection done honestly — the selector belongs inside the loop.

Leak 2: A Column That Arrives After the Answer

Target leakage hides in the data itself. Simulated customer churn with three honest features (tenure, monthly charges, support calls) and one extra column: exit_survey_completed — a survey that only goes to customers who cancelled, plus a few who fill it out anyway.

Features	Test AUC
Honest features only	0.662
Plus exit_survey_completed	0.953

The leaky column turned a mediocre model into a stellar one. In production, nobody has completed an exit survey before leaving, so the feature is empty exactly when you need it. IMO this is the most dangerous kind, because the code is flawless and the score is thrilling.

Ask one question about every column: would I know this value at the moment I make the prediction? If the answer is "only after the outcome," drop it.

Leak 3: Time Doesn't Shuffle

When data has a time order, a random split lets the model peek at neighboring days. Simulated: a relationship that drifts slowly over 1,500 days, with a gradient boosting model given a day-index feature:

Split	R²
Random 80/20	0.825
First 80% train, last 20% test	0.600

With a random split, every test day has training days right next to it, so the model just interpolates. Predicting the future is the real job, and that scored 0.600. Part of that drop is genuine difficulty — the drift includes a steady trend that leaves the training range — but the honest number is the lower one.

Use TimeSeriesSplit, and match your splitter to your data the way the cross-validation guide lays out. Scikit-learn's docs note the default is gap=0; check the API reference if you need a buffer between train and test.

Leak 4: The Same Person on Both Sides

Rows aren't always independent. The setup: 200 patients with 10 visits each. Every patient has a personal "signature" in their lab values, and the label mostly follows the patient's baseline risk.

Validation	AUC
Shuffled K-fold	0.796
GroupKFold	0.495
Oracle that knows each patient's baseline	0.894

A random forest with shuffled folds reached 0.796 by recognizing patients it had already seen. With GroupKFold — which keeps each patient entirely in train or entirely in test — it fell to 0.495. That's chance: the model had learned who the patients were, and nothing about the disease. Scikit-learn's guide describes the same hazard: without grouping, a model can learn person-specific features and look good on people it has already seen.

If your data has repeated customers, patients, devices, or sessions, split by that identifier.

Leak 5: Duplicates

Duplicated rows are leakage in plain sight. Pure noise again (random features, random labels), then a random forest on a clean set versus one where every row appears twice:

Data	Accuracy
No duplicates	0.501
Every row duplicated	0.843

After a random split, most test rows have an identical twin in training, and the forest memorizes it. Deduplicate before splitting, and watch for near-duplicates too — the same product listed twice with slightly different text.

How to Spot Leakage

Leaks are easier to catch than to fix after launch. These checks come from the experiments above:

• Distrust great scores. A churn AUC of 0.95, or a noise-level problem at 84%, deserves suspicion, not a party.
• Audit single columns. Score each column's AUC against the target on its own:

from sklearn.metrics import roc_auc_score

def audit_columns(df, target):
    flags = []
    for col in df.columns:
        if col == target or not df[col].dtype.kind in "fi":
            continue
        try:
            auc = roc_auc_score(df[target], df[col])
        except ValueError:
            continue
        auc = max(auc, 1 - auc)          # direction-agnostic
        flags.append((col, round(auc, 3), "SUSPICIOUS" if auc > 0.85 else ""))
    return sorted(flags, key=lambda t: -t[1])

On the churn data it printed:

Column	Single-column AUC	Flag
exit_survey_completed	0.940	SUSPICIOUS
tenure_months	0.650
support_calls	0.612
monthly_charges	0.560

It caught the leaky column instantly — but it's a blunt tool. It misses leaks that are weaker (say, 0.75) or that only show up in combinations of columns, and a legitimately strong feature can trigger it too. Treat a flag as a question to investigate, not a verdict.

• Check feature importance. One feature dominating a model deserves a look.
• Compare validation schemes. If shuffled CV scores well above time-ordered or grouped CV, you've found a leak or a dependency.
• Re-run with a different seed. Randomness choices affect CV — confirm your result isn't a lucky split.

A real-world version of that last collapse: a credit-churn model showed a beautiful 0.91 AUC in validation, then performed like noise in its first month live. The leak was a "hardship flag" column — a status that support only applies after a customer misses payment, timestamped days before the label it was supposedly predicting. The notebook never looked wrong; the question "when would I know this?" was never asked. The column came out, AUC dropped to an honest 0.68, and the model still shipped — because a number you can trust beats a number you can't.

How to Prevent It

1. Split first. Do it before any preprocessing, as the guide says.
2. Use a Pipeline for every fitted step: scaling, imputing, encoding, selecting, and vectorizing.
3. Match the splitter to the data: TimeSeriesSplit for time, GroupKFold for repeated entities, and plain shuffled K-fold only when rows are truly independent.
4. Deduplicate before splitting.
5. Review every column for "when would I know this?"
6. Keep a final untouched test set. Scikit-learn's guide warns that tuning against a test set lets knowledge "leak" into the model and evaluation metrics.

Common Pitfalls

• Scaling, selecting, or imputing before the split — the sneakiest version, because everything still "works."
• Tuning hyperparameters against the test set until the test score quietly becomes another validation score.
• Random splits on time-ordered data — every test row has neighbors in training.
• Ignoring entity IDs — the same customer, patient, or device in both folds.
• Splitting after deduplication checks — duplicates must go before the split.
• Keeping columns that arrive with or after the outcome.
• Trusting a single validation scheme — compare shuffled, time-ordered, and grouped CV against each other.
• Celebrating a score you can't reproduce with a different seed or splitter.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is data leakage in machine learning?

Data leakage is when your model saw information during training that it won't have at prediction time — scikit-learn's guide defines it exactly that way and warns it produces "overly optimistic performance estimates." The classic forms: preprocessing fitted before the split (feature selection, scaling, imputing on all rows), columns that only exist after the outcome (exit surveys, cancellation flags), random splits on time-ordered data, the same entity appearing in both train and test, and duplicated rows. The tell is a score that collapses in production: 95% AUC in validation, coin-flip in the wild.

How do you detect data leakage in a dataset?

Start by distrusting great scores — a churn AUC of 0.95 or 84% accuracy on noise-level data deserves suspicion. Then audit: score each column's single-feature AUC against the target (the helper in this article flagged exit_survey_completed at 0.940 instantly, though it misses weaker or combination-only leaks and can false-positive on legitimately strong features), check whether one feature dominates importance, compare validation schemes (if shuffled CV beats time-ordered or GroupKFold by a lot, you've found leakage or a dependency), and re-run with a different seed since randomness affects CV.

What is the difference between preprocessing leakage and target leakage?

Preprocessing leakage happens when learned statistics see test data: you select features, scale, or impute on the full dataset before cross-validating, so the transformer picks up test-fold information — pure noise with 10,000 random features scored 0.837 accuracy that way versus 0.501 with the selector inside the pipeline. Target leakage hides in the columns themselves: the value only exists after (or because of) the outcome, like exit_survey_completed for churn, which lifted test AUC from 0.662 to 0.953 while being empty in production exactly when you need it. One is a wrong order of operations; the other is a wrong feature.

How do you prevent data leakage end to end?

Split first, before any preprocessing. Put every fitted step — scaling, imputing, encoding, selecting, vectorizing — inside a Pipeline so fit only ever sees training folds. Match the splitter to the data: TimeSeriesSplit for time (check the gap parameter in the API reference for a buffer), GroupKFold for repeated customers, patients, or devices, and shuffled K-fold only when rows are truly independent. Deduplicate before splitting, review every column with "when would I know this?", and keep a final untouched test set — scikit-learn's guide warns that tuning against a test set lets knowledge leak into your metrics.

Wrapping This Up

Leakage inflated scores in all five setups: 0.837 on noise from early feature selection, 0.953 AUC from a post-outcome column, 0.825 versus 0.600 from ignoring time, 0.796 versus 0.495 from mixing patients, and 0.843 from duplicates. The honest numbers were always lower, and always the ones that matter.

Open your most successful model, list its features, and ask of each one: would I have this at prediction time?

Related Articles

• Data Preprocessing for Machine Learning: Complete Step-by-Step Guide
• Scikit-Learn Cross-Validation Master Class
• Feature Selection Methods: Filter, Wrapper, and Embedded Explained
• Data Validation with Great Expectations: Catch Bad Data Early