Data Preprocessing for Machine Learning: Complete Step-by-Step Guide

Contents

Figure 1: The model gets the credit, but the preprocessing decided the result

Split first, fit on training data only, and the rest of the steps get much easier

Most machine learning projects are won or lost before a model is trained. Preprocessing — turning raw, messy data into something a model can learn from — takes a large share of real project time, and it's where the most damaging silent mistakes happen: leakage that inflates your metrics, encodings that break in production, and fixes that quietly distort the data.

This guide walks through the full sequence in the order you should do it, with a working scikit-learn pipeline at the end that ties it together.

The Golden Rule: Split First, Fit on Training Data Only

Every preprocessing step that learns something from data — a median for imputation, a mean and standard deviation for scaling, category lists for encoding, a target mean for target encoding — must learn it from the training set only. If you compute those statistics on the full dataset and then split, information from your test set has leaked into training, and your evaluation will look better than reality.

The practical way to enforce this is to put every learned step inside a scikit-learn Pipeline or ColumnTransformer, so each cross-validation fold fits them on its own training portion automatically — the same discipline the cross-validation guide applies to model evaluation. Keep that principle in mind through every step below.

Step 1: Understand the Data First

Before transforming anything, look at it:

df.shape
df.dtypes
df.describe(include="all").T
df.isna().mean().sort_values(ascending=False)    # share missing per column
df.duplicated().sum()
df["target"].value_counts(normalize=True)        # class balance

Check for: wrong data types (numbers stored as text, dates as strings), duplicate rows, impossible values (negative ages, future birth dates), inconsistent categories ("NY", "ny ", "New York"), constant columns, columns that look like identifiers, and anything that wouldn't be known at prediction time, which signals target leakage. Write down what you find — preprocessing decisions should follow from observations, not habit.

Step 2: Split the Data Correctly

Choose the split to match how the model will be used:

• Random stratified split for independent rows. Stratify on the target for classification so class proportions match.
• Grouped split when multiple rows belong to the same entity (customer, patient, device). Use GroupShuffleSplit or GroupKFold so one entity never appears in both train and test.
• Time-based split for temporal data. Train on the past and test on the future, with TimeSeriesSplit for cross-validation. Random splits on time series leak the future.

from sklearn.model_selection import train_test_split

X = df.drop(columns="target")
y = df["target"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)

Set the test set aside and don't touch it until your final evaluation.

Step 3: Clean the Basics

Some cleaning is deterministic and row-independent — it doesn't learn anything from the data, so it's safe to apply before the split:

• Fix data types: parse dates, convert numeric strings, standardize booleans.
• Normalize text categories: strip whitespace, unify case, merge obvious variants.
• Remove exact duplicates, after thinking about whether they're errors or genuine repeats.
• Correct units and obvious entry errors, using domain rules.
• Replace impossible values with missing, such as an age of 999.
• Fill missing categories with a constant label like "missing", which learns nothing from the data.

Anything that depends on dataset statistics — filling with a median, or removing rows beyond three standard deviations — belongs inside the pipeline after the split.

A Note on pandas 3.0

If you upgraded pandas, two changes from the 3.0 release (January 2026) affect preprocessing code. String columns are now inferred as a dedicated str dtype instead of the NumPy object dtype, which can break code that checks for the object dtype or the exact missing value sentinel — so df.select_dtypes("object") may no longer find your text columns. The robust fix is to list your column groups explicitly, which is good practice anyway. And Copy-on-Write is now the default and only mode, so chained assignments like df["a"][mask] = 0 no longer modify the original — use df.loc[mask, "a"] = 0 instead.

Step 4: Handle Missing Values

Start by asking why values are missing, since the answer shapes the fix. Data missing completely at random is the easy case. Missingness related to other variables (older customers skip a field) or to the value itself (high earners hide income) carries information, and silently filling it can destroy that signal.

Your options:

• Drop rows or columns, only when missingness is rare or a column is mostly empty. Dropping rows can bias your sample.
• Simple imputation: median for skewed numeric features, mean for roughly symmetric ones, most frequent or a constant for categories.
• Model-based imputation: KNNImputer or IterativeImputer, which can help when features are strongly related, at higher cost and complexity.
• Missing indicators: add a binary column flagging that the value was missing, with SimpleImputer(add_indicator=True). This preserves the information that "this was absent," which is often predictive.
• Skip imputation for models that handle missing values natively, such as scikit-learn's histogram gradient boosting and the major boosting libraries. Check your library version for support.

Never impute using statistics from the test set, and always fit imputers on training data only.

Step 5: Deal With Outliers

Outliers are not automatically errors. A very high income might be a genuine high earner, and deleting it would remove real signal. Approach them in order:

1. Detect: use domain rules first, then statistical flags like the IQR rule or robust z-scores.
2. Investigate: is it a data-entry mistake, a unit error, or a real extreme?
3. Decide: fix errors, keep genuine extremes, and consider capping (winsorizing) or a log transform for heavy-tailed features.

Tree-based models are largely insensitive to outliers, while linear models, k-NN, SVMs, and neural networks can be heavily affected. RobustScaler uses medians and quartiles, which reduces outlier influence on scaling. Avoid deleting rows because of a statistical threshold alone, especially in small datasets or rare-event problems where the "outliers" may be exactly what you're trying to predict.

Step 6: Encode Categorical Features

Models need numbers, and the right encoding depends on the feature:

• One-hot encoding for low-cardinality nominal features. Use handle_unknown="ignore" so new categories at prediction time don't crash the model. Scikit-learn's OneHotEncoder also offers min_frequency and max_categories to group rare categories, and sparse_output to control sparse versus dense output.
• Ordinal encoding for features with a genuine order (small, medium, large), specifying the order explicitly. It's also acceptable for tree models on nominal data, though not for linear ones.
• Target encoding for high-cardinality features like city or zip code, replacing each category with a smoothed average of the target. It's powerful and leakage-prone if done naively. Scikit-learn's TargetEncoder uses cross-fitting to reduce that risk, and its documentation notes that fit(X, y).transform(X) does not equal fit_transform(X, y) because of the cross-fitting scheme — so always use it inside a pipeline.
• Frequency or count encoding as a simple alternative for high-cardinality features.
• Hashing for extreme cardinality, trading interpretability for fixed size.
• Native categorical support in histogram gradient boosting, LightGBM, and CatBoost, which can skip manual encoding.
• Embeddings for deep learning on high-cardinality categories.

Avoid label-encoding nominal categories for linear models, since it invents an order that doesn't exist.

Step 7: Scale and Transform Numeric Features

Not every model needs scaling:

• Needs it: k-NN, SVMs, k-means, PCA, neural networks, and regularized linear models (ridge, lasso, logistic regression with regularization), where feature scale affects distances, gradients, or penalties — as covered in the regularization guide.
• Doesn't need it: decision trees, random forests, and gradient boosting, which only care about feature ordering.

Common transformers: StandardScaler (zero mean, unit variance), MinMaxScaler (a fixed range, sensitive to outliers), and RobustScaler (outlier-resistant). For skewed features, apply a log transform (np.log1p for non-negative data) or use PowerTransformer or QuantileTransformer to make distributions more symmetric. For skewed regression targets, TransformedTargetRegressor applies and inverts a transform automatically.

Step 8: Engineer Features

Good features usually beat clever algorithms. Common moves:

• Dates: extract month, day of week, hour, and elapsed time since an event. Encode cyclical values (month, hour) with sine and cosine so December sits next to January. Use a fixed reference date, not today(), so results are reproducible.
• Ratios and interactions: debt-to-income, price per square foot, and similar domain-driven combinations.
• Binning: KBinsDiscretizer can help linear models capture non-linear effects.
• Aggregations: counts and averages per customer or group, computed carefully using only information available at prediction time.
• Text: TF-IDF vectors, or embeddings from a pretrained model.
• Polynomial features for linear models, used sparingly.

The whole sequence slots into a repeatable pipeline architecture — see the feature engineering pipeline article for the component-by-component build. The cardinal sin here is target leakage: a feature that encodes the outcome or information from the future. If a model's performance looks too good, suspect a leaky feature before celebrating.

Step 9: Select Features and Reduce Dimensions

More features aren't always better. Remove constant and duplicate columns, drop features that can't be known at prediction time, and consider pruning highly correlated groups. Useful tools include variance-threshold filters, L1-regularized models for embedded selection, and permutation importance on held-out data. PCA compresses correlated numeric features but makes them harder to interpret — the PCA tutorial covers that tradeoff in depth. Whatever you use, put selection inside the cross-validation loop, since selecting features on the full dataset leaks information.

Step 10: Handle Class Imbalance (Training Data Only)

For rare-event problems, options include class_weight="balanced", resampling (SMOTE and variants through imbalanced-learn), and threshold tuning, covered in the rare event detection guide in this series. Two rules matter: resample only the training data, never the validation or test data, and use imbalanced-learn's own Pipeline so resampling happens inside each fold. Start with class weights and threshold tuning before generating synthetic samples.

Putting It Together: A Complete Pipeline

This example builds a messy synthetic customer dataset and a single pipeline covering imputation with missing indicators, a log transform, scaling, one-hot and target encoding, and date features. Because everything lives in the pipeline, cross-validation fits all of it on training folds only:

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import (FunctionTransformer, OneHotEncoder,
                                   StandardScaler, TargetEncoder)

# --- Synthetic messy data -------------------------------------------------
rng = np.random.default_rng(0)
n = 3000
df = pd.DataFrame({
    "age": rng.normal(40, 12, n).round(),
    "income": np.exp(rng.normal(10.5, 0.7, n)),
    "plan": rng.choice(["basic", "plus", "pro"], n, p=[0.6, 0.3, 0.1]),
    "city": rng.choice([f"city_{i}" for i in range(200)], n),
    "signup": pd.Timestamp("2022-01-01") + pd.to_timedelta(rng.integers(0, 1000, n), unit="D"),
})
df.loc[rng.random(n) < 0.08, "age"] = np.nan
df.loc[rng.random(n) < 0.05, "income"] = np.nan
df.loc[rng.random(n) < 0.02, "plan"] = np.nan

logit = (-2 + 0.02 * (df["age"].fillna(40) - 40)
         - 0.00001 * df["income"].fillna(df["income"].median())
         + (df["plan"] == "basic") * 0.8)
df["churn"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

# --- Step 3: deterministic cleaning (safe before the split) ----------------
df["plan"] = df["plan"].fillna("missing")        # a constant fill learns nothing from data

# --- Step 2: split ---------------------------------------------------------
X, y = df.drop(columns="churn"), df["churn"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)

# --- Steps 4-8: the preprocessing pipeline --------------------------------
def add_date_parts(frame):
    s = pd.to_datetime(frame["signup"])
    return pd.DataFrame({
        "month_sin": np.sin(2 * np.pi * s.dt.month / 12),
        "month_cos": np.cos(2 * np.pi * s.dt.month / 12),
        "days_since_signup": (pd.Timestamp("2025-01-01") - s).dt.days,   # fixed reference date
    }, index=frame.index)

preprocess = ColumnTransformer(
    [
        ("age", Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
                          ("scale", StandardScaler())]), ["age"]),
        ("income", Pipeline([("log", FunctionTransformer(np.log1p)),
                             ("impute", SimpleImputer(strategy="median", add_indicator=True)),
                             ("scale", StandardScaler())]), ["income"]),
        ("plan", OneHotEncoder(handle_unknown="ignore", min_frequency=0.02), ["plan"]),
        ("city", TargetEncoder(random_state=0), ["city"]),
        ("dates", FunctionTransformer(add_date_parts), ["signup"]),
    ],
    remainder="drop",
    sparse_threshold=0,          # force dense output for models that need it
)

# --- Evaluate with leak-free cross-validation ------------------------------
cv = StratifiedKFold(5, shuffle=True, random_state=0)
for name, clf in {
    "logistic regression": LogisticRegression(max_iter=1000),
    "hist gradient boosting": HistGradientBoostingClassifier(random_state=0),
}.items():
    pipe = Pipeline([("prep", preprocess), ("model", clf)])
    scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="roc_auc")
    print(f"{name:<24} AUC {scores.mean():.3f} ± {scores.std():.3f}")

A few design points are worth noticing. Columns are listed explicitly instead of selected by dtype, which sidesteps the pandas 3.0 string-dtype change. The log transform comes before imputation so missing values pass through unchanged, and the missing indicators are added after. The date function uses a fixed reference date. TargetEncoder runs inside the pipeline, so its cross-fitting happens within each training fold. And the test set stays untouched until you've chosen your final model. If you want readable feature names from the fitted pipeline, give FunctionTransformer a feature_names_out argument, and set_output(transform="pandas") returns DataFrames from compatible transformers.

Productionizing Preprocessing

Save the whole pipeline — preprocessing and model together — with joblib, so serving applies exactly the transformations used in training. Preprocessing code that lives in a notebook and gets reimplemented by hand in production is a major source of train-serve skew. Validate inputs at serving time with schema checks and the data quality tools covered earlier in this series, and monitor feature distributions for drift. Version your pipeline code with your data and model, and add unit tests for custom transformers.

A Short Checklist

1. Inspect the data and define what can be known at prediction time.
2. Choose the split that matches deployment, and set aside the test set.
3. Apply deterministic cleaning.
4. Build a pipeline with imputation, outlier handling, encoding, scaling, and engineered features suited to your model type.
5. Evaluate with cross-validation, then confirm once on the test set.
6. Ship the entire pipeline, and monitor it.

Common Pitfalls

• Fitting scalers, imputers, or encoders on the full dataset before splitting.
• Resampling before the split or before cross-validation.
• Selecting features using all the data.
• Target-encoding without cross-fitting.
• Dropping rows with missing values without checking what you remove.
• Deleting outliers that are genuine and informative.
• Label-encoding nominal categories for linear models.
• Scaling for models that don't need it, or forgetting it for models that do.
• Allowing features that wouldn't exist at prediction time.
• Using random splits on temporal or grouped data.
• Using today() in date features, which makes results irreproducible.
• Hand-rewriting preprocessing for production instead of shipping the pipeline.
• Checking dtype equality to object after upgrading to pandas 3.0.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the correct order of data preprocessing steps?

Split first, always. Then: inspect the data, apply only deterministic cleaning (steps that learn nothing from statistics), handle missing values, deal with outliers, encode categories, scale numerics, engineer and select features, and handle imbalance — with every step that learns anything from data (medians, means, category lists, target means) fit on training data only, inside a pipeline. The test set stays untouched until final evaluation.

How do I prevent data leakage in preprocessing?

Put every learned step — imputers, scalers, encoders, target encoders, feature selection — inside a scikit-learn Pipeline or ColumnTransformer, and evaluate with cross-validation so each fold fits those steps on its own training portion. Never compute statistics on the full dataset before splitting, never resample before the split, and never select features using all the data.

Should I scale features for every machine learning model?

No. Scale for models where feature magnitude affects distances, gradients, or penalties: k-NN, SVMs, k-means, PCA, neural networks, and regularized linear models. Decision trees, random forests, and gradient boosting only care about feature ordering, so scaling changes nothing for them — skipping it saves time and keeps pipelines simpler.

How should I handle missing values in machine learning?

First ask why they are missing. Drop rows or columns only when missingness is rare or a column is mostly empty; otherwise impute — median for skewed numeric features, most frequent or a constant for categories — and add missing indicators with SimpleImputer(add_indicator=True) so the model can use the fact that a value was absent. Fit imputers on training data only, and skip imputation entirely for models that handle missing values natively.

Wrapping This Up

Good preprocessing means making each step suit your data and model, and above all keeping everything that learns from data inside a pipeline fit on training data only. Clean deterministically, impute thoughtfully with missing indicators, treat outliers as questions rather than errors, encode categories according to cardinality and model type, scale only where it matters, engineer features without leaking the target, and let cross-validation tell you whether any of it helped.

Where should you start on your next project? Build the leak-free pipeline skeleton first, with a simple model on top, then improve one step at a time while watching cross-validated results. Small, validated improvements to your data pipeline almost always beat swapping to a fancier algorithm.

Related Articles

• Building a Feature Engineering Pipeline with Scikit-learn Pipelines
• Scikit-learn Cross Validation: Master K-Fold and Stratified Techniques
• PCA Tutorial: Dimensionality Reduction with Python
• Synthetic Data for Imbalanced Classification: SMOTE and Beyond