Scikit-learn Classification Recipes: Raw Data to Evaluation

Contents

Figure 1: Scikit-learn Classification Recipes: Raw Data to Evaluation — inspect, split, pipeline, baseline, validate, tune, evaluate

Your model scores 98% accuracy. You celebrate, ship it, and then it flags zero fraud cases. Sound familiar? I have lived that embarrassment, and it taught me that a good classifier starts long before fit().

This guide walks you through scikit-learn classification recipes from messy raw data to honest evaluation. Every step uses code you can paste into a notebook today. Ready to build a pipeline you can actually trust?

Recipe 1: Load and Inspect Your Data

Look at your data before you touch a model. Check the shape, the column types, the missing values, and especially the class balance. Five minutes here saves five hours of confusion later.

import pandas as pd

df = pd.read_csv('customers.csv')
print(df.shape)
print(df.dtypes)
print(df.isna().sum())
print(df['churned'].value_counts(normalize=True))

That last line matters most. If one class covers 95% of your rows, a lazy model can hit 95% accuracy by guessing the same label every time. Keep that number in mind, because it shapes every choice that follows.

Recipe 2: Split Your Data the Right Way

Split first, then preprocess. This order protects you from data leakage, the sneaky mistake where information from your test set leaks into training. Leakage makes your scores look great and your production results look terrible.

from sklearn.model_selection import train_test_split

X = df.drop(columns='churned')
y = df['churned']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

Notice the stratify=y argument. It keeps the class ratio identical in both splits, which matters a lot with imbalanced data. Set random_state too, so you can reproduce your results tomorrow.

Recipe 3: Build a Preprocessing Pipeline

Real datasets mix numbers and categories, and both need different treatment. A ColumnTransformer lets you handle each type in one tidy object. Then a Pipeline bolts the preprocessing onto your model.

Handle Numeric Columns

Numeric columns often have gaps and wildly different scales. Fill the gaps with the median and scale the values. Scaling matters for models like logistic regression and SVMs, though tree models mostly ignore it.

Handle Categorical Columns

Categorical columns need encoding before a model can read them. Fill missing values with the most common category, then apply one-hot encoding. Set handle_unknown='ignore' so new categories at prediction time do not crash your app.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

num_cols = X_train.select_dtypes(include='number').columns
cat_cols = X_train.select_dtypes(exclude='number').columns

numeric = Pipeline([
    ('impute', SimpleImputer(strategy='median')),
    ('scale', StandardScaler()),
])

categorical = Pipeline([
    ('impute', SimpleImputer(strategy='most_frequent')),
    ('encode', OneHotEncoder(handle_unknown='ignore')),
])

preprocess = ColumnTransformer([
    ('num', numeric, num_cols),
    ('cat', categorical, cat_cols),
])

The pipeline learns the medians, scales and categories from the training data only. It then reuses them on the test data. That single design choice prevents most leakage bugs.

Recipe 4: Start With a Baseline Model

Before you try anything fancy, build a boring baseline. A baseline gives you a number to beat, and it tells you whether your fancy model earns its complexity. Logistic regression works well here because it trains fast and explains itself.

from sklearn.linear_model import LogisticRegression

baseline = Pipeline([
    ('prep', preprocess),
    ('model', LogisticRegression(max_iter=1000)),
])

baseline.fit(X_train, y_train)
print(baseline.score(X_test, y_test))

I always start here. Half the time, the simple model lands close enough that I stop and move on to the next problem. Why overbuild when a straight line does the job?

Recipe 5: Try Stronger Classifiers

Now bring in the heavier models. Swap the final step of your pipeline and keep everything else the same. That is the beauty of pipelines: you change one line and compare fairly.

Model	How it works	Best when
Random forest	Averages many decision trees	Messy features, little tuning time
Gradient boosting	Builds trees in sequence, each fixing the last one's mistakes	Most tabular problems; HistGradientBoostingClassifier is fast and handles missing values natively
Support vector machine	Finds a clean boundary between classes	Smaller datasets
K-nearest neighbours	Labels a point by its closest neighbours	Simple problems, small data

Gradient boosting wins most tabular problems. Random forests come second and need less fuss. Neural networks rarely beat either one on plain spreadsheet data.

from sklearn.ensemble import RandomForestClassifier

forest = Pipeline([
    ('prep', preprocess),
    ('model', RandomForestClassifier(n_estimators=300, random_state=42)),
])

Recipe 6: Use Cross-Validation, Not One Lucky Split

A single train-test split can flatter or punish your model by pure luck. Cross-validation trains and tests on several different splits, then averages the results. You get a score you can trust and a sense of how much it wobbles.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(forest, X_train, y_train, cv=5, scoring='f1')
print(scores.mean(), scores.std())

Pay attention to the standard deviation. A high average with a big wobble means your model behaves unpredictably. Cross-validation exposes that before your users do.

Recipe 7: Tune Hyperparameters

Tuning squeezes extra performance out of a good model. GridSearchCV tries every combination in a grid you define, while RandomizedSearchCV samples combinations at random and finishes faster. Start with the randomized version when your grid gets large.

from sklearn.model_selection import GridSearchCV

params = {
    'model__n_estimators': [200, 400],
    'model__max_depth': [None, 10, 20],
}

search = GridSearchCV(forest, params, cv=5, scoring='f1', n_jobs=-1)
search.fit(X_train, y_train)
print(search.best_params_)

The double underscore tells scikit-learn which pipeline step owns each parameter. Tune on the training data only, and leave the test set alone until the very end. Peeking early defeats the whole point.

Recipe 8: Handle Imbalanced Classes

Imbalanced data breaks naive models. When one class dominates, the model learns to ignore the rare one. Two quick fixes help you recover:

1. Class weights: set class_weight='balanced' on models that support it. The model then penalizes mistakes on the rare class more heavily.
2. Threshold tuning: models usually label a case positive above 0.5 probability. Lower that cutoff when you care more about catching rare cases.

Both tricks cost almost nothing to try. Resampling methods exist too, but start with weights and thresholds. They solve most problems without extra libraries.

Recipe 9: Evaluate With the Right Metrics

Accuracy lies when classes are imbalanced. Use metrics that match your real-world costs. Here is the short list I rely on:

Metric	Question it answers
Precision	Of the cases you flagged, how many were truly positive?
Recall	Of all truly positive cases, how many did you catch?
F1 score	The balance between precision and recall in one number
ROC AUC	How well the model ranks positives above negatives across all thresholds
Confusion matrix	Exactly where the model gets confused

Which metric should you pick? Ask what hurts more: a false alarm or a missed case. A cancer screening test cares about recall, while a spam filter cares about precision.

from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score

best = search.best_estimator_
preds = best.predict(X_test)
probs = best.predict_proba(X_test)[:, 1]

print(classification_report(y_test, preds))
print(confusion_matrix(y_test, preds))
print(roc_auc_score(y_test, probs))

Run this on the test set exactly once. That score estimates how your model will perform on new data, so treat it with respect.

Common Pitfalls

I have made every one of these. Learn from my scars.

1. Preprocessing before splitting. This leaks information and inflates your scores.
2. Trusting accuracy alone. Check precision, recall and the confusion matrix.
3. Tuning on the test set. Use cross-validation for tuning and save the test set for the final check.
4. Skipping the baseline. Without a simple benchmark, you cannot judge your fancy model.
5. Forgetting to save the pipeline. Save the whole pipeline with joblib, not just the model, so preprocessing travels with it.
6. Fitting the encoder on the full dataset. OneHotEncoder must see training categories only, or new-category handling stops meaning anything.
7. Reporting a single split score. It is one sample, not an estimate.
8. Ignoring the standard deviation from cross-validation. A mean without a spread hides an unstable model.

Wrapping Up

A reliable classifier follows a repeatable recipe. Inspect your data, split first, build a pipeline, set a baseline, cross-validate, tune, handle imbalance, and judge with the right metrics. Skip a step and the model will punish you later.

So grab a dataset and run through these recipes once, start to finish. Then change one thing and see what happens. Trust me, the 98% accuracy that catches nothing is a lesson you only need to learn once.

When you are ready to put a model behind an API, validating tool call outputs with Pydantic covers the boundary where model output becomes an action, and securing an MCP server covers the credentials if an agent will call your model. For LLM-side classification, function calling vs tool calling explains the interface.

Frequently Asked Questions

Why does my classifier show 98% accuracy but catch nothing?

Because accuracy measures the majority class, not the class you care about. If 98% of rows are negative, a model that always predicts negative scores 98% and finds zero positives. Check the class balance before you train, then judge the model on precision, recall and the confusion matrix for the rare class. That is the failure this guide's recipe order is designed to prevent.

Should I preprocess before or after train_test_split?

After. Split first, then fit every preprocessing step on the training data only. If you impute, scale or encode before splitting, statistics from the test set leak into training and your score stops estimating performance on new data. A scikit-learn Pipeline enforces the correct order automatically, because the transformers only see the training fold during fit.

When should I use a Pipeline instead of manual preprocessing?

Always, once the code leaves a notebook. A Pipeline keeps imputation, scaling and encoding attached to the model, so the same transformations are applied at prediction time and cross-validation cannot leak between folds. It also makes hyperparameter search cleaner, because you tune preprocessing and model settings through one grid using step names. Save the whole pipeline with joblib, not just the estimator.

Which classifier works best for tabular data?

Gradient boosting wins most tabular problems, and HistGradientBoostingClassifier is the fast option on larger datasets because it handles missing values natively. Random forests come second and need less tuning. Logistic regression is still the right baseline, because it trains fast, explains itself, and sometimes lands close enough that you can stop there. Neural networks rarely beat these on plain spreadsheet data.

How do I handle imbalanced classes in scikit-learn?

Start with class_weight='balanced' on models that support it, then tune the decision threshold instead of accepting the default 0.5. Both are cheap and need no extra libraries. Resampling methods exist, but weights and thresholds solve most problems first. Whatever you choose, evaluate with recall and precision rather than accuracy, because accuracy hides the rare class you are trying to catch.

How many times should I evaluate on the test set?

Once, at the very end. Every peek at the test score turns it into a tuning signal, which means it stops being an unbiased estimate of performance on new data. Tune with cross-validation on the training data, compare models there, and leave the held-out set for the single final measurement you report.

Sources

• [1] scikit-learn, "Pipelines and composite estimators" — Pipeline, ColumnTransformer, and why transformers must be fitted inside the pipeline so test data never touches training statistics: scikit-learn.org/modules/compose
• [2] scikit-learn, "Preprocessing data" — StandardScaler, OneHotEncoder and its handle_unknown behaviour: scikit-learn.org/modules/preprocessing
• [3] scikit-learn, "Imputation of missing values" — SimpleImputer strategies, including median and most-frequent fills: scikit-learn.org/modules/impute
• [4] scikit-learn, "Ensemble methods" — random forests, gradient boosting and HistGradientBoostingClassifier with native missing-value support: scikit-learn.org/modules/ensemble
• [5] scikit-learn, "Metrics and scoring" plus the model-selection API — classification_report, confusion_matrix, roc_auc_score, train_test_split with stratify, cross_val_score and GridSearchCV: scikit-learn.org/modules/model_evaluation, train_test_split

Related Articles

• Validating Tool Call Outputs with Pydantic
• Function Calling vs Tool Calling: What Actually Differs
• Best LLM APIs for Tool Calling and Function Calling
• Securing an MCP Server: OAuth, Permissions, Least Privilege