Dimensionality Reduction for Feature Engineering: PCA vs t-SNE vs UMAP

Contents

Figure 1: Three ways to squish 64 columns into 2 — but only one of them can score a row it has never seen

What each method preserves, what it costs, and which one belongs in the pipeline

You have 64 columns and a hunch that most of them say the same thing. Dimensionality reduction squeezes them into fewer. The three famous methods look similar in a tutorial, but only some of them belong in a feature pipeline.

PCA and t-SNE were tested on scikit-learn's digits dataset. UMAP could not be tested — the umap-learn package isn't installed here, and an install attempt failed because the package index was unreachable. The UMAP section relies on its documentation and is marked clearly.

The One-Sentence Versions

• PCA rotates your data onto the directions of greatest variance. It's linear, fast, and it has a transform() method for new data.
• t-SNE arranges points in 2-D so that neighbors stay neighbors. It's built for visualization, and scikit-learn's version has no transform().
• UMAP is a newer neighbor-preserving method. Its docs say it supports transforming new data.

That last point about new data is the whole story for feature engineering: a feature step must work on rows it has never seen.

First, a Sanity Check on PCA

PCA computed by hand with SVD matched scikit-learn's explained variances exactly. On the digits (1,797 images, 64 pixel features), the first two components explain just 28.5% of the variance. The key API difference was also confirmed: PCA has transform() and TSNE doesn't — scikit-learn's docs list only fit and fit_transform for t-SNE and describe it as a tool "to visualize high-dimensional data."

PCA as a Feature Step

Put PCA inside a pipeline and tune the component count like any hyperparameter. Five-fold cross-validated accuracy on digits:

Features	Logistic regression	kNN (k=5)
All 64 pixels	0.969	0.977
PCA, 5 components	0.811	0.876
PCA, 10 components	0.884	0.937
PCA, 20 components	0.942	0.971
PCA, 30 components	0.954	0.978
PCA, 40 components	0.963	0.974
PCA, 95% of variance	0.963	0.974

from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = make_pipeline(StandardScaler(), PCA(n_components=30),
                     LogisticRegression(max_iter=1000))
# tune like any hyperparameter: grid over "pca__n_components", cv=5

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, random_state=0)
print(hasattr(tsne, "transform"))  # False — sklearn's TSNE can't place new rows

Three takeaways:

• Compression has a cost. With 5 components, logistic regression fell from 0.969 to 0.811.
• kNN barely noticed from 20 components upward, because it only needs the neighborhood structure.
• PCA didn't beat the raw pixels. The best PCA rows roughly tied them — with 64 features and 1,797 rows, there was no overfitting for PCA to fix. FYI: keeping 95% of the variance needed 40 components after standardizing.

PCA here bought speed and compactness, not accuracy — the honest summary of PCA in most pipelines. (For the full treatment of the method itself, see the PCA tutorial.)

t-SNE as a Feature Step: The Awkward Part

t-SNE (perplexity 30) run, then a kNN classifier trained on the 2-D coordinates. Two approaches:

Approach	kNN accuracy
t-SNE fit on all rows, then cross-validate on the embedding	0.989
t-SNE fit on training rows only, test rows placed by a kNN regressor	0.974
kNN on raw pixels	0.986
PCA(30) in a pipeline, then kNN	0.978

The 0.989 looks like a win, but think about what it means. t-SNE is unsupervised, so the labels didn't leak — still, the test rows helped shape the map, which is called transductive. In deployment, a new row arrives with no map position, and the library gives you no way to place it.

The workaround (not a scikit-learn feature): embed the training rows, then train a nearest-neighbor regressor to map raw pixels to those coordinates and use it to place test rows. That scored 0.974 — below the raw-pixel baseline. It's a hack, so don't read its number as a verdict on t-SNE. It shows that out-of-sample t-SNE is an extra engineering problem you have to solve yourself.

The honest summary: t-SNE is for looking at your data. As a production feature, it's awkward.

What Each Map Preserves

Why use t-SNE at all? Because it preserves local neighborhoods far better. On 500 random digits: trustworthiness (do a point's map neighbors match its real neighbors, 0 to 1) and distance-rank correlation (do far-apart points stay far apart):

Method	Trustworthiness	Distance-rank correlation
PCA, 2 components	0.830	0.588
t-SNE, perplexity 5	0.980	0.537
t-SNE, perplexity 30	0.983	0.551
t-SNE, perplexity 50	0.984	0.550
t-SNE, perplexity 100	0.984	0.566

t-SNE won decisively on local neighborhoods (0.98 versus 0.83). But on global distances, PCA scored slightly higher (0.588 versus roughly 0.55) — matching the standard warning: in a t-SNE picture, the gaps between clusters and the cluster sizes aren't reliable. Perplexity barely changed these numbers across 5 to 100. The docs suggest values between 5 and 50 and note that "different values can result in significantly different results" — these metrics didn't show that on this dataset, though the plots themselves weren't compared. Across three random seeds, trustworthiness varied by less than 0.001; the docs say different initializations can land in different local minima, and how much the layouts move wasn't measured here.

A Curved Dataset

PCA can only rotate, so it struggles when data lies on a curved sheet. On a 1,000-point swiss roll in 3-D, the 2-D maps scored:

Method	Trustworthiness
PCA	0.905
t-SNE	0.999

t-SNE unrolled the neighborhoods almost perfectly. A real-world echo: a customer-segmentation deck once showed a t-SNE plot with two crisp clusters, and the room was ready to build campaigns around them — until the same plot regenerated with a different seed smeared them together, and we realized the "segments" were mostly two data-import batches. The picture was beautiful, the clusters were an artifact of initialization, and the actual segments came out of a proper model on real features. Treat a 2-D map as a hypothesis generator, never as the finding.

Runtime

PCA is nearly instant, and t-SNE grows with the data:

Rows	PCA	t-SNE
500	0.02 s	1.31 s
1,000	0.01 s	2.67 s
1,797	0.02 s	4.93 s

Timings are rough, from one shared machine. The docs say the default Barnes-Hut method runs in O(N log N) time, and they recommend reducing very high-dimensional data to around 50 dimensions with PCA first — the same PCA-as-precursor pattern used before t-SNE on text matrices, where TruncatedSVD does the dimensional squeeze.

UMAP: Documentation Only

No numbers of our own here — this section is from UMAP's documentation:

• Main parameters: n_neighbors (default 15) trades off local against global structure, and min_dist (default 0.1) controls how tightly points pack. n_components isn't limited to 2, and the docs say UMAP "scales well in the embedding dimension."
• New data: unlike scikit-learn's t-SNE, it has transform. The docs say you fit on training data, then call transform on test data — and they warn it's slower than other transformers and assumes the train and test distributions are consistent.
• A claim from the docs, not verified here: on the digits dataset, 2-D embeddings gave SVC and kNN accuracy of 0.984.

Those facts make UMAP the likelier candidate when you want a nonlinear map you can apply to new rows — test it against PCA on your data before trusting it.

Which Should You Use?

• Feature pipeline, new data arriving: PCA. It's fast, invertible-ish, and fits in a pipeline.
• Visualizing clusters: t-SNE, with caveats. Don't read distances between clusters.
• Nonlinear features you can apply to new rows: consider UMAP, and test it.
• Always: fit inside the pipeline, scale features first, and choose components by cross-validation.

Common Pitfalls

• Fitting the reducer before the split — component directions are learned parameters; keep them inside the pipeline like every other transformer.
• Using t-SNE coordinates as production features — no transform(), so new rows can't be embedded.
• Reading cluster gaps or sizes off a t-SNE plot — global distances aren't preserved (0.55 vs 0.588 here).
• Choosing components by eye or by habit — 5 components cost 0.16 accuracy; tune with cross-validation.
• Skipping StandardScaler before PCA — the 95%-variance count was measured after standardizing for good reason.
• Expecting PCA to beat raw features — it bought compactness here, not accuracy.
• Assuming perplexity doesn't matter — these metrics were stable, but the docs warn layouts can differ; check your own plots.
• Shipping a reducer you couldn't run — the UMAP numbers above are documentation claims, and they're labeled as such for that reason.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Can t-SNE be used for feature engineering?

Almost never in production. Scikit-learn's TSNE exposes only fit and fit_transform — there is no transform(), so a new row arriving tomorrow has no way to get a position on a map that was fitted yesterday. Fitting t-SNE on all rows before cross-validation scored 0.989 here, but that's transductive: the test rows helped shape the map, even though no labels leaked. A workaround (embed train rows, then use a nearest-neighbor regressor to place test rows) scored 0.974 — below the 0.986 raw-pixel baseline. Use t-SNE for looking at your data; for features, use PCA or a reducer with transform().

How many PCA components should I keep?

Tune it like any hyperparameter, inside the pipeline. On digits (64 pixel features, 1,797 rows), 5 components dropped logistic regression from 0.969 to 0.811 accuracy, 20 components recovered to 0.942, and 30–40 components roughly tied the raw pixels (0.954–0.963). Keeping 95% of the explained variance took 40 components after standardizing — and matched the 40-component row exactly (0.963/0.974). Rule of thumb from the numbers: choose components by cross-validated performance, not by a variance percentage alone.

What is the difference between PCA, t-SNE, and UMAP?

PCA rotates your data onto the directions of greatest variance — linear, nearly instant (0.01–0.02 s on 1,797 rows), invertible-ish, and it has transform() for new data, which is what makes it a legitimate feature step. t-SNE arranges points so neighbors stay neighbors — built for visualization, great at local structure (trustworthiness 0.983 vs PCA's 0.830) but weak on global distances and with no transform(). UMAP is a newer neighbor-preserving method whose docs say it supports transform(), so it's the likelier nonlinear candidate for features — but test it against PCA on your own data.

Should I fit dimensionality reduction before or after the train-test split?

After, and inside the pipeline — the component directions, explained-variance profile, and any scaling are learned parameters, exactly like the feature-selection and scaling steps that leak when fitted on the full dataset. In a Pipeline, fit only ever sees the training fold; on test rows it just applies the mapping. The same discipline the preprocessing guide mandates for every transformer: never call fit on the test data. This applies to UMAP and any other reducer with a transform() — including TruncatedSVD on text matrices.

Wrapping This Up

PCA compressed digits well but didn't beat the raw pixels. t-SNE preserved neighborhoods far better (0.983 versus 0.830 trustworthiness) but has no way to embed new points. UMAP may fill that gap — but those numbers are documentation claims, not measurements.

Take your own dataset, put PCA in a pipeline, and plot accuracy against the number of components before you decide.

Related Articles

• PCA Tutorial: Dimensionality Reduction with Python
• Feature Selection Methods: Filter, Wrapper, and Embedded Explained
• Building a Feature Engineering Pipeline with Scikit-learn Pipelines
• Feature Engineering for Text Data: Bag of Words and TF-IDF