Naive Bayes Classifier: Text and Tabular Examples in Python

Contents

Figure 1: Count the words, multiply the evidence, take the winner

Two from-scratch builds, exact scikit-learn matches, and a measured cost of "naive"

Naive Bayes is the classifier everyone calls "too simple to work." Then it filters your spam, scores your sentiment, and embarrasses a neural network on a small dataset.

The name tells you the catch. The "naive" part is a deliberately wrong assumption, and the model works anyway. Below, we build Naive Bayes from scratch twice — once for text and once for numeric data — and match both against scikit-learn. We'll also measure what the naive assumption actually costs. Every number comes from a script run with scikit-learn 1.9.1.

The Idea in Plain English

Naive Bayes uses Bayes' rule to ask: given what I see, which class is most likely?

P(class | features) ∝ P(class) × P(feature₁ | class) × P(feature₂ | class) × ...

Two pieces matter:

• The prior, P(class): how common each class is.
• The likelihoods, P(feature | class): how typical each feature is for that class.

The "naive" part is multiplying the likelihoods together as if features were independent given the class. Real features rarely are. In an email, "free" and "prize" show up together far more than chance would predict. But the model only needs the right class to score highest, not the right probability, and that's much easier.

Text Example: Spam vs. Ham

An honest note about the data first. I couldn't download a standard text corpus in my sandbox, so I wrote 40 short messages (20 spam, 20 ham) by hand. That's enough to show the mechanics and verify the code, but not enough to make claims about real-world accuracy. Use your own data for that.

Turning Text Into Numbers

Naive Bayes for text typically works on word counts, a "bag of words":

from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer()
X = vec.fit_transform(texts)      # one row per message, one column per word

My 40 messages produced a vocabulary of 159 words.

The From-Scratch Version

This is the entire multinomial classifier:

import numpy as np

class ScratchMultinomialNB:
    def __init__(self, alpha=1.0):
        self.alpha = alpha

    def fit(self, X, y):
        self.classes_ = np.unique(y)
        self.log_prior_ = np.log(np.array([(y == c).mean() for c in self.classes_]))
        counts = np.array([X[y == c].sum(axis=0) for c in self.classes_]) + self.alpha
        self.log_lik_ = np.log(counts / counts.sum(axis=1, keepdims=True))
        return self

    def joint_log_proba(self, X):
        return X @ self.log_lik_.T + self.log_prior_

    def predict(self, X):
        return self.classes_[self.joint_log_proba(X).argmax(axis=1)]

Two details deserve attention.

• Logs, not products. Multiplying hundreds of tiny probabilities underflows to zero. Adding their logarithms doesn't. So the code sums log P(word | class) instead.
• The alpha term. Adding alpha to every word count is called Laplace smoothing. We'll see in a moment why it's essential.

Checking Against Scikit-learn

from sklearn.naive_bayes import MultinomialNB

model = MultinomialNB(alpha=1.0).fit(X_train, y_train)

I trained on 28 messages and tested on 12. Both versions made identical predictions, with probabilities differing by at most 1.4e-15 (floating-point dust). Test accuracy was 0.833 for both, which is 10 of 12 correct. Twelve test messages can't tell you much about quality, but they do confirm the code is right.

A Message, Worked Through

For "claim your free prize now," the model computes a log score for each class:

• Ham: -26.387
• Spam: -18.863

Spam wins, with a probability of 0.9995. Notice how that happens. Each word adds a small nudge to one class or the other, and five spammy words add up fast.

Which Words Matter?

Subtract the log likelihoods and you get each word's pull. The strongest spam-leaning words: now (2.14), free (2.14), money, click and claim (1.45 each). The strongest ham-leaning: the (-1.64), at (-1.55), can (-1.32), you (-1.19) and meeting (-1.03).

That's a nice property. Naive Bayes is easy to inspect. You can read off why it decided what it did.

Why Smoothing Exists

Here's the trap. If a word never appears in one class during training, its probability there is zero. Multiply by zero and the whole score collapses, no matter how much other evidence you have.

I tested it with the message "your prize is meeting." The first three words are the strongest spam-leaning words seen in both classes. The last word, "meeting," never appears in spam training data:

alpha	P(spam)
1e-10 (almost no smoothing)	0.000000
0.1	0.460622
1.0	0.693511

With almost no smoothing, a single unseen word vetoes spam outright. With smoothing, the model weighs all four words. FYI, that's why scikit-learn defaults to alpha=1.0.

Choosing Alpha

I scored four alpha values with 5-fold cross-validation repeated 10 times, with the vectorizer inside the pipeline so it never sees held-out data — the setup from the cross-validation guide in this series:

alpha	Accuracy
0.01	0.900 ± 0.090
0.1	0.917 ± 0.081
1.0	0.925 ± 0.087
3.0	0.902 ± 0.101

Every difference sits inside the spread. On 40 messages, alpha barely matters. Don't over-tune it.

Tabular Example: Gaussian Naive Bayes

For numeric features, swap word counts for Gaussian (bell-curve) likelihoods. For each class and feature, the model stores a mean and a variance, then asks how plausible a new value is under that curve.

class ScratchGaussianNB:
    def fit(self, X, y, var_smoothing=1e-9):
        self.classes_ = np.unique(y)
        self.mean_ = np.array([X[y == c].mean(0) for c in self.classes_])
        self.var_ = np.array([X[y == c].var(0) for c in self.classes_])
        self.var_ += var_smoothing * X.var(0).max()   # same variance floor as scikit-learn
        self.log_prior_ = np.log(np.array([(y == c).mean() for c in self.classes_]))
        return self

    def joint_log_proba(self, X):
        ll = -0.5 * (np.log(2 * np.pi * self.var_)[None]
                     + (X[:, None, :] - self.mean_[None]) ** 2 / self.var_[None])
        return ll.sum(axis=2) + self.log_prior_

On scikit-learn's wine dataset (178 wines, 13 features, 3 classes; 70/30 split), my version and GaussianNB both scored 0.9630, with identical predictions and probabilities matching to 2e-15. I left the features unscaled, because Gaussian Naive Bayes models each feature separately, so scaling doesn't matter the way it does for k-NN.

What "Naive" Really Costs

Let's measure the independence assumption instead of just asserting it. I repeated every wine feature n times. That makes the features perfectly dependent — the worst case for a model that assumes independence:

Copies of each feature	Test accuracy	Mean max probability	Log loss
1	0.9630	0.9918	0.0704
2	0.9630	0.9981	0.1286
5	0.9630	1.0000	0.3225
10	0.9630	1.0000	0.6489

Accuracy never moved. But confidence ran away. The model counts the same evidence again and again, so it grows more certain without learning anything new, and the log loss climbs from 0.07 to 0.65. This is the real lesson: Naive Bayes ranks classes well but its probabilities are often overconfident. In my experience that's the one fact to remember. Use the labels freely. Be careful with the probabilities.

When to Use Naive Bayes

It's a strong choice when:

• You need a fast, tiny baseline, especially for text.
• You have little data. It needs few parameters.
• You want to explain a decision by pointing at features.

Think twice when:

• You need trustworthy probabilities. Calibrate them, or use another model.
• Features are strongly redundant, as the experiment above shows.
• Feature interactions drive the answer.

Compared with the other classifiers in this series, Naive Bayes trades calibration for speed and small-data robustness — and as the logistic regression guide shows, a linear model on the same features gives you honest probabilities at the cost of a real training step.

Common Pitfalls

• Skipping smoothing — one unseen word can veto an entire class.
• Multiplying probabilities directly instead of working in log space, risking underflow.
• Tuning alpha hard on tiny data, where every difference sits inside the noise.
• Trusting raw probabilities without checking calibration.
• Assuming the independence assumption is harmless — duplicated or correlated features inflate confidence.
• Fitting the vectorizer on all data before splitting, which leaks held-out vocabulary into training.
• Using Gaussian NB on unscaled data is fine, but not k-NN — know which models care about scale.
• Dismissing Naive Bayes as too simple before benchmarking it on your own problem.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Why is Naive Bayes called "naive"?

Because it multiplies the per-feature likelihoods together as if the features were independent given the class — an assumption real data almost never satisfies. In an email, "free" and "prize" co-occur far more often than chance. The trick is that the model only needs the right class to score highest, not the right probability, and getting the ranking right is much easier than getting the calibration right.

Why does the code work in logarithms instead of multiplying probabilities?

Multiplying hundreds of tiny probabilities underflows floating-point arithmetic to exactly zero, which destroys the comparison between classes. Adding logarithms never underflows and turns the product into a sum, so the implementation computes X @ log_likelihood.T + log_prior instead. It's mathematically identical to Bayes' rule but numerically stable.

What is Laplace smoothing and why does alpha matter?

Laplace smoothing adds alpha to every word count before normalizing, so no word ever gets a probability of exactly zero. Without it, a single word unseen in a class during training vetoes that class completely: with alpha=1e-10 a test message scored P(spam)=0.000000, while alpha=1.0 gave 0.69. scikit-learn defaults to alpha=1.0 for exactly this reason, and cross-validation showed the exact value barely matters on small data.

Are Naive Bayes probabilities reliable?

Usually not without calibration. Because the model multiplies evidence it treats as independent, duplicated or correlated features make it grow more certain without learning anything new: duplicating every wine feature left accuracy at 0.9630 but pushed log loss from 0.07 to 0.65 as the mean max probability hit 1.0000. Trust the ranking, calibrate the probabilities with something like CalibratedClassifierCV if you need them numerically.

Wrapping This Up

Naive Bayes multiplies simple per-feature evidence under an assumption that's wrong and often useful. It matches scikit-learn when you smooth counts and work in logs, it's easy to read, and its probabilities need a grain of salt.

Next step: grab the script, swap in your own labeled text, and rerun it. Then try the duplicate-features trick on your data and see how overconfident it gets.

Related Articles

• K-Nearest Neighbors Tutorial: Simple Classification in Python
• Logistic Regression Explained: Build a Classifier Step by Step
• Machine Learning Algorithms Explained: Complete Beginner's Guide
• Scikit-learn Cross-Validation: Master K-Fold Testing