Handling Skewed Data: Log, Box-Cox, and Yeo-Johnson Transforms

Contents

Figure 1: The long right tail of real numbers — income, prices, wait times — is where means, plots, and linear models go wrong

Five transforms, known shapes, and a 21% error hiding in plain sight

Income, prices, page views, wait times: real-world numbers love a long right tail. A few huge values drag the average, stretch your plots, and make linear models fit the wrong thing.

The standard fix is a transform. But "just take the log" skips three problems: zeros, negatives, and what happens when you convert predictions back. The usual options were tested on data with known shape, and one result shows a 21% error you'd never notice by accident.

What Skew Does, and What Transforms Do

Skewness measures asymmetry: zero means symmetric, and big positive numbers mean a long right tail. Transforms squeeze the tail so the data looks more bell-shaped. These are the main tools:

• Log: np.log(x). Needs strictly positive values. log1p computes log(1 + x) and handles zeros.
• Square root: gentler than log, needs non-negative values.
• Box-Cox: a family of power transforms with a fitted parameter λ (lambda). λ=0 is the log, λ=1 leaves data unchanged. Requires strictly positive data.
• Yeo-Johnson: like Box-Cox, but works on zero and negative values. It's the default in scikit-learn's PowerTransformer.
• Quantile transform: maps values to a normal shape by rank. Scikit-learn's docs note it "distorts correlations and distances within and across features."

Scikit-learn picks λ by maximum likelihood, separately for each feature. Both formulas were checked by hand against PowerTransformer(standardize=False) — they match.

Round One: How Much Skew Gets Removed?

Four skewed datasets (5,000 values each, 20 repeats), measuring skewness after each transform. "n/a" means the transform can't handle that data:

Data	Raw	Log	Log1p	Sqrt	Box-Cox	Yeo-Johnson	Quantile
Lognormal	5.01	-0.00	1.13	1.69	0.00	0.13	-0.00
Exponential	1.97	-1.14	0.43	0.62	-0.04	0.06	-0.00
30% zeros + lognormal	6.43	n/a	0.57	1.08	n/a	0.11	-0.71
Lognormal minus 3 (negatives)	6.11	n/a	n/a	n/a	n/a	0.48	-0.00

Four lessons jump out:

• Match the transform to the shape. Log is perfect for lognormal data by definition (0.00), but it overshoots exponential data (-1.14), flipping the skew to the left. Box-Cox fixed that case (-0.04) because it can choose a milder λ.
• Zeros and negatives break the simple options. Box-Cox raised an error on zeros: "can only be applied to strictly positive data."
• Yeo-Johnson handled everything, with residual skew from 0.06 to 0.48 — not perfect on the shifted-lognormal case (0.48).
• The quantile transform stumbled on ties. With 30% exact zeros, it left skewness at -0.71, because tied values can't be spread out by rank.

IMO, Yeo-Johnson is the safest default when you don't know your data's range, and plain log is fine when you do.

A λ You Shouldn't Over-Read

Fitted lambdas for the same lognormal data: Box-Cox gave -0.029 (near 0, which is the log, as expected). Yeo-Johnson gave -0.895. That looks wildly different, but it isn't a contradiction — Yeo-Johnson on positive data behaves like Box-Cox applied to x + 1, and a lognormal with lots of mass near zero feels that shift. So don't compare λ values across the two methods. FYI: scikit-learn's docs also warn that power transforms work very well on some distributions and are "ineffective" on others, so plot before and after.

Does It Help the Model? It Depends on the Relationship

A lognormal feature with linear models on 2,000 test rows (20 datasets). First case: the true relationship is y = 3·log(x) + noise:

Feature handling	Linear-model test R²
Raw x	0.494
log x	0.901
Yeo-Johnson	0.887
Quantile	0.898
HistGradientBoosting on raw x	0.892

Transforming nearly doubled R². Then the truth was flipped: y = 2·x + noise, a plainly linear relationship:

Feature handling	Linear-model test R²
Raw x	0.951
log x	0.548
Yeo-Johnson	0.517
Quantile	0.559
HistGradientBoosting on raw x	0.837

Now every transform wrecked the fit. A transform doesn't just fix a histogram — it changes the relationship your model sees. Skewed features aren't automatically a problem; make the transform match how the target actually depends on the feature. The tree model landed in the middle both times (0.892 and 0.837), so it's a decent hedge when you don't know.

The Target Is Different: Back-Transforming Bites

Skewed targets are the common case in regression, and the usual move is to model log(y) and apply exp to predictions. Scikit-learn's TransformedTargetRegressor does this: per its docs, it fits on func(y) and predicts with inverse_func.

Here's the trap. Simulated y = exp(1 + 0.8x + noise) with noise sd 0.7, scored on the original scale (100 datasets):

Method	RMSE	MAE	Mean prediction ÷ mean actual
Linear on raw y	5.701	3.150	1.003
Log target, naive exp()	5.341	2.469	0.787
Log target + smearing	5.159	2.624	1.006
Yeo-Johnson target	5.866	2.497	0.780

The naive log model predicted averages about 21% too low. That matches theory: exponentiating a log-scale mean gives roughly the median, which sits below the mean by about exp(-σ²/2), or 0.783 here. The fix used is "smearing": multiply predictions by the average of exp(training residuals):

import numpy as np

# fit on log(y), predict on log scale, then smear back
log_pred = model.predict(X_test)
resid = np.exp(y_train_log - model.predict(X_train))  # exp of training residuals
smear = resid.mean()
y_pred = np.exp(log_pred) * smear                      # unbiased-ish mean on original scale

It brought the ratio to 1.006 and gave the best RMSE.

But look at MAE. The naive log model had the best MAE (2.469), because medians are good at minimizing absolute error. So "which is better" depends on your metric, and on whether you need the mean (totals, revenue forecasts) or the typical value.

A real-world version of that trap: a monthly revenue forecast built on a log target looked perfectly reasonable in back-testing — until someone summed the predictions for the quarter and compared against finance's totals, where the naive exp() version ran about a fifth low every single month. Nobody noticed for two cycles because each individual prediction "looked fine"; only the aggregate exposed the bias. If you report sums or totals, check the mean-prediction ratio, not just RMSE.

Failure Modes to Expect

Two practical problems showed up.

New data can break Box-Cox. Box-Cox was fit on positive training data, then passed a zero. It raised the same ValueError. Yeo-Johnson transformed that zero to -2.257 without complaint. If production data might contain zeros or negatives, Box-Cox is a time bomb.

Extreme values move λ. A few values multiplied by 1,000 were added to 1,000 lognormal points (averages of 30 datasets):

Extreme values added	Box-Cox λ	Yeo-Johnson λ
0	0.007	-0.847
1	-0.056	-0.852
5	-0.145	-0.862
20	-0.237	-0.856

With 20 extreme points (2% of the data), Box-Cox's λ drifted from about 0 to -0.24. Yeo-Johnson's barely moved in this test, though that pattern isn't explained by these runs and shouldn't be generalized. Either way, a λ fitted on dirty data is a λ you should inspect — related to the outlier question the outlier guide covers: decide whether those extreme points are errors before letting them steer the transform.

Which Transform Should You Use?

• Strictly positive, roughly lognormal: plain log. It's simple and interpretable.
• Positive, unsure how skewed: Box-Cox, if you can guarantee positivity forever.
• Zeros, negatives, or unknown ranges: Yeo-Johnson, or log1p for counts.
• Skewed target: transform it, but evaluate on the original scale and decide whether you need a mean or a median. Use smearing or a similar correction for means.
• Tree models: they don't need any of this. Trees split on order, not on scale.
• Always: fit the transformer on training data only, put it in a pipeline (the same rule as scaling and every other learned step in the preprocessing guide), and plot before and after.

Common Pitfalls

• Taking the log of data with zeros or negatives "somehow" — log1p or Yeo-Johnson instead of ad-hoc constants.
• Exponentiating log-target predictions and reporting the mean — the 21% bias is silent in RMSE and loud in totals.
• Fitting Box-Cox on training positives and meeting a zero in production — it's a ValueError waiting to happen.
• Over-reading λ — Box-Cox and Yeo-Johnson lambdas aren't comparable, and dirty data moves them.
• Transforming because the histogram looks bad, not the relationship — it halved R² when the truth was linear.
• Using the quantile transform on heavily tied data — ranks can't separate exact zeros.
• Forgetting trees don't care — a monotone transform changes nothing about their splits.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Which transform should I use for skewed data?

Match it to the data's range and shape. Strictly positive and roughly lognormal: plain log — it hit 0.00 skew by definition. Positive but unsure how skewed: Box-Cox, if you can guarantee positivity forever. Zeros, negatives, or unknown ranges: Yeo-Johnson (scikit-learn's PowerTransformer default) or log1p for counts. Skewed target: transform it, but evaluate on the original scale and decide whether you need the mean (use smearing) or the median. Tree models need none of this — they split on order, not scale.

Why does Box-Cox fail with zeros or negative values?

Because its formula is only defined for strictly positive data — in these tests it raised ValueError: can only be applied to strictly positive data, both on zeros during fitting and when a zero arrived at prediction time after training on positives. Yeo-Johnson, which is like Box-Cox but handles zero and negative values, transformed that same zero without complaint. If production data might contain zeros or negatives, Box-Cox is a time bomb; Yeo-Johnson or log1p are the safe defaults.

Why are my log-transformed predictions too low?

Because exponentiating a log-scale mean gives roughly the median, not the mean. In the simulated regression, the naive log-target model's predictions averaged 0.787 of the true mean — a 21% under-prediction matching the theory of exp(-σ²/2) ≈ 0.783 for the noise level used. The fix is smearing: multiply predictions by the average of exp(training residuals), which brought the ratio to 1.006 and gave the best RMSE. If you optimize MAE instead of RMSE, the naive median-flavored prediction may actually be what you want.

Do tree models need skewed features transformed?

No. Trees split on order, not on scale, so a monotone transform doesn't change which splits they choose. In these tests, HistGradientBoosting on raw features landed in the middle both times — 0.892 when the truth was logarithmic (where transforms reached ~0.90) and 0.837 when the truth was linear (where transforms fell to ~0.52-0.56). That makes a tree model a decent hedge when you don't know how the target depends on the feature, but the transform question simply doesn't apply to it.

Wrapping This Up

Transforms can fix skew, but they aren't free. Log overshoots exponential data, Box-Cox rejects zeros, and quantile transforms stumble on ties. A transform helped a linear model when the truth was logarithmic and hurt badly when it was linear. And back-transforming a log model under-predicted the mean by 21% until it was corrected.

Take one skewed column of your own, plot it before and after two transforms, and check the model's score on the original scale.

Related Articles

• Feature Scaling Explained: StandardScaler vs MinMaxScaler vs RobustScaler
• Outlier Detection and Removal in Python: IQR, Z-Score, and Isolation Forest
• Data Preprocessing for Machine Learning: Complete Step-by-Step Guide
• Building a Feature Engineering Pipeline with Scikit-learn Pipelines