Encoding Categorical Variables: One-Hot, Label, and Target Encoding

Contents

Figure 1: Words in, numbers out — the encoding you pick quietly decides what the model is allowed to learn

Three encoders, three promises — here's which ones keep them

Models like numbers. Your data has words: colors, cities, job titles, zip codes. Somewhere between the CSV and .fit(), you have to turn those words into numbers, and the way you do it can quietly decide whether your model works.

Three standard approaches were tested on synthetic data, so the truth was controlled and each method's recovery measurable. One result was dramatic. Another was a lot less clear-cut than the usual advice suggests.

The Three Encoders in One Minute

• Label (ordinal) encoding replaces each category with an integer: red=0, blue=1, green=2. In scikit-learn that's OrdinalEncoder. (LabelEncoder exists too, but it's meant for target labels, not input features.)
• One-hot encoding makes one 0/1 column per category. That's OneHotEncoder.
• Target encoding replaces each category with a (shrunk) average of the target for that category. That's TargetEncoder, added in scikit-learn 1.3.

Each one makes a different promise. Let's see which ones keep it.

Label Encoding: Dangerous for Unordered Categories

Integer codes imply an order and a spacing. Is blue "between" red and green? For a linear model, the answer is yes, whether you meant it or not.

Six unordered categories with true effects of 0, 3, -2, 4, -3, and 1 were built alongside a numeric feature and noise. Then a linear model (Ridge with a tiny penalty) was fit on each encoding, across 20 random datasets:

Encoding	Test R²
Integer codes	0.363 (sd 0.040)
One-hot	0.910 (sd 0.010)

The ceiling for this setup is about 0.911, so one-hot essentially recovered everything. Integer codes got a third of the way there — the linear model tried to draw one straight line through effects that zigzag, and it couldn't.

IMO, this is the most common categorical mistake. Label encoding isn't wrong — it's wrong for linear models on nominal data. Use it for truly ordered categories (small, medium, large) or with tree models that can split the codes into pieces.

One-Hot Encoding: Safe, With Two Catches

One-hot is the default for good reason. It makes no ordering assumption, and the linear model above shows what that buys you. The scikit-learn docs list these defaults: drop=None, sparse_output=True, handle_unknown='error'.

Two catches matter in practice:

• Width. A feature with 3,000 categories becomes 3,000 columns. That's fine for a sparse linear model and awkward for everything else.
• Unseen categories. Tested against a category the encoder never saw: the default raises a ValueError at prediction time; handle_unknown='ignore' returns all zeros, so the model treats it as a blank.

FYI: the drop option removes one column per feature. The docs say that helps when perfectly collinear features cause problems, such as an unregularized linear regression — and they warn that dropping a category can introduce bias in penalized models, so don't reach for it by reflex.

Target Encoding: Powerful and Easy to Misuse

Target encoding squeezes a category into one column. It solves the width problem, which is why people love it for zip codes and product IDs.

The raw version is simple: replace each category with the mean target for that category. That has an obvious problem — a category seen twice gets a noisy mean, and a category seen once gets exactly its own target value. The model then learns to trust a column built from the answer.

Scikit-learn's version fixes this with shrinkage. The docs describe it as mixing the global target mean with the category's own mean. The standard formula verified against the library:

encoded = (n_c * category_mean + m * global_mean) / (n_c + m)

where n_c is the category's count and m is the smoothing strength. It matched scikit-learn's output for every category. In the test, a rare category with 25 rows had a raw mean of 6.070; after shrinkage it became 4.804, pulled toward the global mean of 1.639. Rare categories get pulled harder, which is the point.

The second fix is cross-fitting. According to the docs, fit_transform encodes each training row using only the other folds, which is "the recommended method for encoding training data." Calling fit(X, y).transform(X) is discouraged because it "can introduce data leakage." In code:

from sklearn.preprocessing import TargetEncoder

enc = TargetEncoder(target_type="continuous", cv=5)
X_train_enc = enc.fit_transform(X_train, y_train)  # cross-fitted
X_test_enc = enc.transform(X_test)                 # full-train encodings

Also set target_type="continuous" for regression — the docs warn that integer targets can be mistaken for multiclass. And unseen categories get the global target mean, which was confirmed (4.5 in the toy case).

The Experiment: 200 Categories

Now the real question. Two hundred categories with skewed counts (a few common, many rare), 3,000 training rows and 3,000 test rows, averaged over 8 datasets. True category effects and noise each have variance 4, so the best possible test R² is about 0.5.

Method	Train R²	Test R²
One-hot + Ridge	0.501	0.433
Integer codes + HistGB (numeric)	0.477	0.411
Integer codes + HistGB (native categorical)	0.378	0.343
Naive target mean + Ridge	0.506	0.423
Naive target mean + HistGB	0.506	0.424
TargetEncoder (cross-fitted) + Ridge	0.422	0.429
TargetEncoder (cross-fitted) + HistGB	0.472	0.392
TargetEncoder (fit, then transform) + HistGB	0.505	0.424

Read it carefully, because it doesn't say what the folklore says:

• One-hot won. With 15 rows per category on average, plain one-hot with Ridge scored best on test.
• The naive encoder did fine. With enough rows per category, leakage barely hurt.
• Cross-fitting did what it promises, but at a price. Its train score (0.422) is honest and close to its test score (0.429), while the naive version's train score (0.506) overstates reality. Yet cross-fitting with HistGB scored lower on test (0.392) than the naive version (0.424) — not investigated further here.
• HistGB's native categorical mode scored lowest. Defaults without tuning, so no general conclusion.

When Target Encoding Earns Its Keep

The 200-category setup was too comfortable. So training size was held at 3,000 rows while the category count increased, scoring test R² with Ridge:

Categories	Rows per category	One-hot	Naive mean	TargetEncoder	Unseen test rows
50	60	0.444	0.444	0.445	0.0%
200	15	0.432	0.425	0.430	0.2%
1,000	3	0.359	0.314	0.339	9.7%
3,000	1	0.304	0.257	0.285	23.8%

Now the story shows up. As categories get rarer, the naive encoder falls behind sharply (0.257 vs 0.285 for the cross-fitted TargetEncoder at 3,000 categories), because it leaks — shrinkage plus cross-fitting recovers most of that gap.

But look at the one-hot column: it never lost. In these tests, TargetEncoder beat the naive version but not one-hot with Ridge. Its real advantage is size: one column instead of thousands, which matters when memory, model type, or speed rules out one-hot.

That last point hit home on a real project: a customer-support dataset had a ticket_subject_freeform column with roughly 230,000 unique values, and the obvious OneHotEncoder expansion produced a feature matrix with hundreds of thousands of columns — enough to stall the training job on memory alone before a single model was fit. Switching that column to a smoothed target encoding (with cross-fitting inside the pipeline) collapsed it to one feature and made the run possible. High cardinality doesn't just widen your matrix; at scale it can stop the job entirely.

Which Encoder Should You Use?

• Few categories, unordered (under a few dozen): one-hot. It was the best or tied for best in every test here.
• Truly ordered categories: ordinal encoding with the correct order passed in explicitly.
• Tree models: try integer codes, since trees can split them — numeric HistGB scored 0.411 against 0.433 for one-hot, so check on your data.
• Very high cardinality: TargetEncoder with fit_transform on training data only. Don't write your own unsmoothed version.
• Anything headed to production: decide what happens to unseen categories before launch. The options ranged from an error to a blank to the global mean.

Whatever you pick, keep it inside a Pipeline with the rest of your preprocessing, as the preprocessing guide shows — and pair it with the scaling choice from the scaler comparison for the numeric columns.

Common Pitfalls

• Label-encoding nominal categories into a linear model — the order you never specified gets invented for you.
• Calling fit(X, y).transform(X) on a TargetEncoder — the docs themselves call it leakage.
• Hand-rolling an unsmoothed target mean — rare categories memorize their own targets.
• Ignoring unseen categories until serving time — the default raises an error the first time production sees a new value.
• One-hotting a 100k-cardinality column — you'll learn about it from your memory monitor, not your metrics.
• Forgetting target_type="continuous" for regression — scikit-learn may treat your integer target as multiclass.
• Assuming native categorical support is automatically better — in this test it scored lowest.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Should I use label encoding for categorical features?

Not for unordered (nominal) categories going into a linear model. Integer codes imply an order and a spacing the data doesn't have — in this experiment, integer codes scored a test R² of 0.363 versus 0.910 for one-hot on the same data. Label encoding is correct for truly ordered categories (small, medium, large) with the order passed explicitly, and it's often fine with tree models that can split the codes into pieces.

When should I use one-hot encoding?

For most cases: few to moderate categories, unordered data, any model. One-hot made no ordering assumption and was best or tied for best in every test here, including 200 categories (0.433 test R² with Ridge). Its two catches are width — 3,000 categories become 3,000 columns — and unseen categories, where the default raises a ValueError at prediction time unless you set handle_unknown='ignore', which returns all zeros.

What is target encoding and why is it cross-fitted?

Target encoding replaces each category with the average of the target for that category, which compresses high-cardinality columns into one. The raw version leaks: a category seen once gets exactly its own target value, so the model learns from the answer. Scikit-learn's TargetEncoder fixes this two ways — shrinkage pulls rare categories toward the global mean, and cross-fitting (fit_transform, not fit().transform()) encodes each training row using only the other folds, which the docs call the recommended method for encoding training data.

Which encoding is best for high-cardinality features?

TargetEncoder with fit_transform on training data only. In the rare-category sweep it produced the best test score among encoders at 3,000 categories (0.285), against 0.257 for a naive leaked mean. Don't write your own unsmoothed version, set target_type="continuous" for regression, and decide what happens to unseen categories before launch — they receive the global target mean.

Wrapping This Up

Integer codes wrecked a linear model on unordered categories (R² 0.363 vs 0.910 with one-hot). One-hot is the safe default but gets wide. Target encoding compresses high-cardinality columns, and smoothing plus cross-fitting keeps it honest — but it didn't beat one-hot accuracy in these tests.

Pick one categorical column in your data today, count its unique values, and match it to the guide above.

Related Articles

• Feature Scaling Explained: StandardScaler vs MinMaxScaler vs RobustScaler
• Data Preprocessing for Machine Learning: Complete Step-by-Step Guide
• Building a Feature Engineering Pipeline with Scikit-learn Pipelines
• Best Feature Engineering Books and Courses