PCA Tutorial: Dimensionality Reduction Explained with Python

Contents

Figure 1: Find the axes where the data actually spreads

From-scratch SVD, variance cutoffs, and the case where PCA throws away the signal

You have 64 features and no idea which ones matter. Plotting them is impossible, training on them is slow, and half of them probably say the same thing in different words.

Principal component analysis (PCA) handles this. It finds the directions along which your data varies most, then lets you keep only those. Below, we build PCA from scratch, match it against scikit-learn, and test where it helps and where it quietly misleads. Every number comes from a script run with scikit-learn 1.9.1.

What PCA Does

Picture a cloud of points shaped like a tilted cigar. The long axis carries most of the information about where each point sits. The short axes carry little. PCA finds those axes and ranks them:

• Principal components (PCs): new axes, each a weighted mix of your original features.
• Order: PC1 captures the most variance, PC2 the most of what's left, and so on.
• Orthogonal: each axis is perpendicular to the others, so they carry no overlapping variance.

Keep the first few PCs and drop the rest, and you've reduced dimensions while keeping most of the spread. The key phrase is "most of the variance." We'll see why that wording matters.

PCA From Scratch

PCA takes three steps: center the data, take its singular value decomposition (SVD), and read off the axes.

import numpy as np

class ScratchPCA:
    def __init__(self, n_components):
        self.n_components = n_components

    def fit(self, X):
        self.mean_ = X.mean(axis=0)
        Xc = X - self.mean_                                    # 1. center the data
        U, S, Vt = np.linalg.svd(Xc, full_matrices=False)      # 2. SVD of the centered data
        var = S ** 2 / (len(X) - 1)                            # variance along each axis
        self.components_ = Vt[: self.n_components]             # rows = principal directions
        self.explained_variance_ = var[: self.n_components]
        self.explained_variance_ratio_ = var[: self.n_components] / var.sum()
        return self

    def transform(self, X):
        return (X - self.mean_) @ self.components_.T

    def inverse_transform(self, Z):
        return Z @ self.components_ + self.mean_

About fifteen lines. The SVD does the heavy lifting, and it's numerically steadier than computing a covariance matrix and its eigenvectors by hand.

Checking Against Scikit-learn

from sklearn.decomposition import PCA

pca = PCA(n_components=5).fit(X)

I tested on the standardized wine dataset (178 wines, 13 features), reducing to 5 components. The variance ratios matched to four decimals:

Component	Share of variance
PC1	0.3620
PC2	0.1921
PC3	0.1112
PC4	0.0707
PC5	0.0656

The axes themselves agreed to 8.9e-16. One catch worth knowing: the sign of each principal axis is arbitrary. Flip it and you get an equally valid answer, so compare absolute values, as the script does. FYI, this trips up plenty of people who see "different" results between libraries.

Two components capture 55.4% of the variance, and five capture 80.2%.

Gotcha #1: Scale Your Features

PCA chases variance, and variance depends on units. Raw wine features differ wildly: the largest variance is 98,609.6 and the smallest is 0.0154. Here's what happens without scaling:

Features	PC1 share of variance	Top PC1 loading
Raw	0.9981	proline (1.000)
Standardized	0.3620	flavanoids (0.423)

On raw data, PC1 is just the proline column, and it "explains" 99.8% of the variance only because proline's numbers are large. Nothing about the wines is being summarized. After standardizing, PC1 becomes a real blend, led by flavanoids (0.423), total phenols (0.395) and a diluted-wine absorbance measure (0.376). Standardize first, unless your features already share units.

How Many Components to Keep?

Use the digits dataset: 1,797 handwritten digits, each an 8×8 image of 64 pixels. The cumulative explained variance tells you how many components cover a target:

Variance to keep	Components needed (of 64)
80%	13
90%	21
95%	29
99%	41

You can let scikit-learn pick for you by passing a fraction:

PCA(n_components=0.95)      # keeps enough components for 95% of the variance

It chose 29, matching the table. Note these figures use the raw pixel values, with no scaling. Three of the 64 pixels are constant in every image, so they carry no variance at all.

What You Lose: Reconstruction Error

Compressing then expanding back shows what's discarded. Mean squared error per pixel:

Components	MSE	Variance kept
2	13.4210	0.2851
5	8.5424	0.5450
10	4.9143	0.7382
20	1.9843	0.8943
30	0.7681	0.9591
40	0.2215	0.9882
64	0.0000	1.0000

Error drops fast at first, then slowly. That's the usual shape, and it's why a "knee" in the variance curve is a sensible place to stop.

Does PCA Help a Classifier?

A common hope is that fewer features means a better model. I tested it: standardize, apply PCA, then logistic regression, using 5-fold cross-validation with PCA fit inside each fold so nothing leaks — the honest-CV pattern from the cross-validation guide:

Components	CV accuracy
2	0.5348
5	0.7713
10	0.8403
20	0.8993
30	0.9065
64	0.9199
No PCA	0.9204

PCA did not improve accuracy here. With 30 components the model scored 0.9065, about 1.4 points below using all 64 pixels (0.9204). So PCA bought compression, not accuracy. In my experience that's the honest framing: use PCA to speed things up, visualize, or remove redundancy, not as a free accuracy boost. Note this section standardized the pixels first, so its components differ slightly from the unscaled table above.

Gotcha #2: High Variance Isn't the Same as Useful

PCA never looks at your labels. It keeps what varies most, whether or not that varies with the class. To show the failure, I built a two-feature dataset where:

• Feature A is loud noise (spread 10) unrelated to the class.
• Feature B is quiet (spread 0.5) but shifted by the class.

PC1 explained 98.67% of the variance, and its direction was [1.000, 0.009]: essentially pure Feature A, the noise. Logistic regression on both original features scored 0.9810. After PCA to one component, accuracy fell to 0.5530, barely above the 0.5 you'd get by guessing.

PCA kept the loudest axis and threw away the one that mattered. If you have labels and want directions that separate classes, look at supervised methods such as linear discriminant analysis instead — see the algorithms guide for the wider family.

When to Use PCA

It works well for:

• Visualization: squash data to 2 or 3 axes to look for structure.
• Compression and speed: fewer features, faster models.
• Decorrelating redundant features.
• Noise reduction, when noise lives in the low-variance directions.

Be careful when:

• Features are on different scales and you haven't standardized — the same trap as k-NN.
• The signal is subtle and low-variance, as in the failure above.
• You need to explain results to others. PCs are blends, and "PC3" means little to a stakeholder.
• Structure is nonlinear. PCA only finds straight axes, though I didn't test a nonlinear case here.

A Practical Checklist

1. Standardize your features.
2. Plot cumulative variance and pick a target such as 90% or 95%.
3. Fit PCA inside your pipeline so cross-validation stays honest.
4. Compare the model with and without PCA.
5. Check the loadings. Do the top components make sense?
6. Remember the sign is arbitrary.

Common Pitfalls

• Running PCA on unscaled features and letting the widest-ranging variable own PC1.
• Assuming fewer components means better accuracy — measure it.
• Fitting PCA before the train/test split, which leaks test variance into the components.
• Reading PC1 as "the most important feature" — it's a blend, and it's only variance-important.
• Choosing a variance target without checking reconstruction error or your downstream task.
• Ignoring the arbitrary sign when comparing results across libraries.
• Using PCA before a classifier when labels matter — supervised decomposition exists for that.
• Dropping components that carry a subtle but discriminative signal.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

How many PCA components should I keep?

Pick a cumulative variance target — 90% or 95% is common — and take the number of components that reaches it. On the digits dataset (64 pixel features) that meant 21 components for 90%, 29 for 95%, and 41 for 99%. You can pass the target directly with PCA(n_components=0.95), which chose 29 to match. Then verify against your actual goal: the reconstruction error table shows the diminishing returns, and a model comparison shows whether the compression costs accuracy.

Do I need to scale features before PCA?

Yes, if your features have different units or ranges, because PCA chases variance and variance depends on units. On raw wine features the largest variance was 98,609.6 and the smallest 0.0154, so PC1 became a copy of the proline column explaining 99.8% of variance while summarizing nothing. After standardization PC1 turned into a real blend (flavanoids 0.423, total phenols 0.395) and explained 36.2%. Standardize first unless your features already share units — pixels in the digits experiment were an exception where all features share the same scale.

Does PCA improve model accuracy?

Not inherently — it buys compression, not accuracy. In the digits experiment, logistic regression with 30 PCA components scored 0.9065 cross-validated accuracy versus 0.9204 using all 64 pixels, so PCA cost about 1.4 points while cutting features by half. Use PCA to speed up training, visualize high-dimensional data, or remove redundancy, and always compare the model with and without it rather than assuming fewer features means better performance.

Why can PCA make a classifier worse?

Because PCA never looks at your labels — it keeps what varies most, whether or not that varies with the class. In the two-feature failure case, a loud noise feature (spread 10) drowned out a quiet but class-informative one (spread 0.5): PC1 explained 98.67% of variance but pointed at [1.000, 0.009], essentially pure noise, and accuracy collapsed from 0.9810 to 0.5530 after reducing to one component. If you need directions that separate classes, use supervised methods like linear discriminant analysis instead.

Wrapping This Up

PCA finds the directions of greatest variance, and you can build it in fifteen lines. It matches scikit-learn exactly once you account for sign, it needs scaled features, and it compresses without promising better accuracy. Most important, it keeps what varies, not what matters.

Next step: grab the script and run it on your own dataset. Check how many components cover 95% of your variance, then see what happens to your model's accuracy at that cutoff.

Related Articles

• Logistic Regression Explained: Build a Classifier Step by Step
• K-Nearest Neighbors Tutorial: Simple Classification in Python
• Machine Learning Algorithms Explained: Complete Beginner's Guide
• Scikit-learn Cross-Validation: Master K-Fold Testing