K-Nearest Neighbors Tutorial: Simple Classification in Python

Contents

Figure 1: Ask the closest examples, take the majority

From-scratch NumPy, exact scikit-learn match, and four measured traps

K-nearest neighbors (k-NN) might be the most honest algorithm in machine learning. It doesn't learn a formula or fit weights. To classify something new, it finds the most similar examples it has seen and copies their answer.

That simplicity makes it a perfect first classifier. It also hides some traps, and I hit all of them while writing this. Below, we build k-NN from scratch with NumPy, match it against scikit-learn, and test what breaks it. Every number comes from a script run with scikit-learn 1.9.1.

How k-NN Works

The whole algorithm fits in four steps:

1. Store the training data. (That's the "training.")
2. For a new point, measure its distance to every stored point.
3. Pick the k closest ones — its neighbors.
4. Let them vote. The most common label wins.

Think of asking your five nearest neighbors which restaurant to try. You don't derive a theory of taste. You just go with the majority.

Two choices define the model: how you measure distance (usually Euclidean, the straight-line kind) and the value of k.

The Data

We'll use scikit-learn's wine dataset: 178 wines, 13 chemical measurements each, three grape varieties as classes (59, 71 and 48 samples). I split it into 124 training and 54 test samples, keeping the class mix equal in both.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

scaler = StandardScaler().fit(Xtr)          # fit on training data only
Ztr, Zte = scaler.transform(Xtr), scaler.transform(Xte)

Notice the scaler. We'll see in a moment why it matters so much.

k-NN From Scratch

Here's the entire classifier:

import numpy as np

def knn_predict(X_train, y_train, X_new, k=5):
    preds = []
    for x in X_new:
        dists = np.sqrt(((X_train - x) ** 2).sum(axis=1))   # distance to every training point
        nearest = np.argsort(dists, kind="stable")[:k]       # indices of the k closest
        votes = np.bincount(y_train[nearest])                # count labels among them
        preds.append(votes.argmax())                         # majority vote
    return np.array(preds)

Eight lines. Each loop computes distances, sorts, takes the top k and counts votes.

Checking It Against Scikit-learn

The library version takes two lines:

from sklearn.neighbors import KNeighborsClassifier

model = KNeighborsClassifier(n_neighbors=5).fit(Ztr, ytr)
predictions = model.predict(Zte)

On the 54 test wines, both versions scored 0.9630 accuracy. More importantly, their predictions were identical on every sample, which the script asserts. So the from-scratch code is doing exactly what the library does. FYI, that match isn't guaranteed in general — tie-breaking differs between implementations, so identical output here is a nice result, not a law.

Trap #1: Feature Scaling

Look at the raw feature ranges. One wine measure, proline, spans 278 to 1,680, a range of about 1,400. Another spans roughly 0.5 in width. Euclidean distance treats them equally in raw units, so the big-number feature drowns out everything else.

I ran k-NN twice, once on raw features and once on standardized ones:

Features	Test accuracy
Raw	0.7222
Standardized	0.9630

That's a 24-point gap from one line of preprocessing. Always scale before k-NN. It's the single most common reason people say k-NN "doesn't work."

Trap #2: Choosing k

Small k means jumpy, noisy decisions. Large k means smooth decisions that can blur real boundaries. Don't guess. Use cross-validation on the training data, following the k-fold guide in this series:

from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

scores = []
for k in range(1, 31):
    pipe = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k))
    scores.append(cross_val_score(pipe, Xtr, ytr, cv=5).mean())

Putting the scaler inside the pipeline matters. Each cross-validation fold then fits its own scaler, so no information leaks between folds.

Here's a sample of the results:

k	CV accuracy
1	0.9197
3	0.9277
5	0.9440
10	0.9437
20	0.9597
30	0.9677

The best value was k = 22, at 0.9760. But six values (22, 24, 25, 26, 27 and 29) tied at that score. So treat this as a plateau, not a peak. On 124 training rows, a difference of one sample moves accuracy by about one percentage point. Anything in that range is noise. Note too that the scores trend up with k here, which is common on small, well-separated data.

With k = 22, test accuracy was 0.9815 — one wrong wine out of 54.

A Warning About Small Test Sets

Here's something that surprised me. With k = 1, the model scores 1.0000 on its own training data, because every point is its own nearest neighbor. That's pure memorization. Yet its test accuracy was also 1.0000, better than k = 22.

So does k = 1 win? No. With only 54 test samples, a single test split is noisy, and cross-validation (which averages five splits) ranked k = 1 as the weakest option. In my experience this is a great reminder: pick k from cross-validation, never from the test set, and don't over-read one small split.

Trap #3: Voting Style

By default, every neighbor gets one equal vote. You can instead weight votes by closeness:

KNeighborsClassifier(n_neighbors=22, weights="distance")

Closer neighbors count more. Here, it didn't help:

Voting	CV accuracy
Uniform	0.9760
Distance-weighted	0.9680

Don't assume the fancier option wins. Test it.

Trap #4: Irrelevant Features

k-NN trusts every feature equally. It can't tell which ones matter. So junk features hurt — a lot. I appended columns of pure random noise to the scaled wine data and measured test accuracy, averaged over 20 random draws:

Noise features added	Test accuracy
0	0.9630
10	0.9630 ± 0.0211
50	0.8713 ± 0.0436
200	0.6907 ± 0.0580

Ten junk features barely mattered. Fifty cost nearly nine points. Two hundred dropped accuracy to about 0.69. This is a taste of the curse of dimensionality: as features pile up, distances between points become less meaningful, and "nearest" stops meaning "similar." If your data has many features, select features or reduce dimensions first.

When k-NN Shines (and When It Doesn't)

It works well when:

• You have small to medium data with a handful of meaningful features.
• You want a no-training baseline in five minutes.
• Decision boundaries are irregular, and you don't want to assume a shape.

It struggles when:

• You have many features, especially irrelevant ones.
• You have huge datasets. Every prediction compares against every stored point, so prediction slows as data grows. I didn't benchmark speed here, so treat that as general guidance, not a measurement.
• You can't scale features sensibly, such as with mixed categorical and numeric columns.

A Quick Checklist

Before you trust a k-NN model, run through this:

1. Scale the features.
2. Choose k by cross-validation, inside a pipeline.
3. Compare against a simple baseline, like logistic regression.
4. Check feature quality. Drop noise.
5. Distrust tiny test sets. One sample can swing results by about two points here.

Common Pitfalls

• Fitting the scaler on all data before splitting, which leaks test information into training.
• Picking k from the test set, which turns your evaluation into a tuning loop.
• Judging k-NN on raw, unscaled features — the most common reason it "doesn't work".
• Adding features without checking them, since k-NN can't ignore irrelevant ones.
• Assuming distance weighting is automatically better.
• Trusting one small test split when cross-validation disagrees.
• Using k-NN on very large datasets without considering prediction cost.
• Skipping a baseline comparison — a simple logistic regression may beat it with far less care.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Why does k-NN need feature scaling?

Because it measures Euclidean distance, and raw units decide which features dominate. On the wine dataset one feature (proline) spans about 1,400 units while another spans roughly 0.5, so the big-number feature drowns out everything else: test accuracy was 0.7222 on raw features versus 0.9630 after standardization — a 24-point gap from one line of preprocessing. Fit StandardScaler on training data only, ideally inside the cross-validation pipeline so no fold sees the test fold.

How do I choose the value of k?

Cross-validation on the training data, never the test set. Sweep k over a range inside a pipeline that scales each fold, then read the results as a plateau rather than a peak: on the wine data six values of k tied for the best score, and with only 124 training rows a single sample moves accuracy by about a point, so differences in that range are noise. A larger k smooths jumpy decisions; a smaller k follows local structure.

Should I weight neighbors by distance?

Only if you test it. Setting weights="distance" makes closer neighbors count more, which sounds strictly better, but on the wine dataset it lowered cross-validation accuracy from 0.9760 to 0.9680. Don't assume the fancier option wins — run both and let cross-validation decide, the same way you would tune any other hyperparameter.

Why does k-NN collapse when I add noise features?

k-NN trusts every feature equally and cannot tell signal from junk, so irrelevant columns corrupt the distance metric. In the experiments here, 10 random noise features barely mattered (0.9630), 50 cost nearly nine points (0.8713), and 200 dropped accuracy to about 0.69. This is the curse of dimensionality: as features pile up, distances concentrate and "nearest" stops meaning "similar", so feature selection or dimensionality reduction comes first.

Wrapping This Up

k-NN is a vote among similar examples, and you can write it in eight lines. It matches scikit-learn exactly when the data is scaled. The algorithm's simplicity is also its weakness: it can't ignore bad features, it needs sensible distances, and it's easy to fool yourself with a lucky test split.

Next step: grab the script and try your own data. Remove the scaler and watch accuracy fall. Add noise columns and watch it collapse. Then fix both. You'll never forget these lessons.

Related Articles

• Machine Learning Algorithms Explained: Complete Beginner's Guide
• Logistic Regression Explained: Build a Classifier Step by Step
• Scikit-learn Cross-Validation: Master K-Fold Testing
• Random Forest Algorithm Explained with Python Code