Feature Engineering for Text Data: Bag of Words and TF-IDF

Contents

Figure 1: Words on a page are the raw material — a vectorizer is the machine that turns them into something a model can multiply

Counts, weights, and where lexical features stop working

Machine learning models do math, and text isn't numbers. Before any classifier can read a review, an email, or a support ticket, you have to turn words into a numeric representation.

Two classic techniques do that job: Bag of Words, which counts words, and TF-IDF, which weights those counts by how distinctive each word is. They're simple enough to explain on a napkin, and they remain strong baselines for text classification, search, and clustering — even in an era of embeddings and large language models. This guide covers how they work, how to tune them in scikit-learn, a complete classification example, and where they run out of road.

From Text to Numbers

The overall recipe is the same for every classical text pipeline:

1. Preprocess the text: lowercase it, strip noise, and split it into tokens.
2. Build a vocabulary: the list of distinct tokens across your training documents.
3. Represent each document as a vector with one position per vocabulary term.
4. Feed those vectors to a standard model such as logistic regression, a linear SVM, or naive Bayes.

Bag of Words and TF-IDF differ only in step 3: what number goes in each position.

Preprocessing: Clean, But Don't Over-Clean

Common steps, each of which is a decision rather than a default:

• Lowercasing merges "Great" and "great" and shrinks the vocabulary. Skip it if case carries meaning, as in some entity or acronym tasks.
• Tokenization splits text into words. The default in scikit-learn keeps tokens of two or more word characters, which silently drops single-character tokens and splits contractions and hyphenated words. Languages written without spaces, and domains like code or social media, need custom tokenizers.
• Stop word removal drops very common words like "the" and "of." It can reduce noise, but it can also remove useful signal, and scikit-learn's documentation cautions that the built-in English list has known issues — consider a custom list, or rely on TF-IDF and max_df to down-weight common words instead.
• Stemming and lemmatization reduce "running," "runs," and "ran" to a common form. They shrink the vocabulary at the cost of extra processing and occasional errors.
• Special tokens: numbers, URLs, emails, emoji, and hashtags can be normalized (replace URLs with a placeholder token), kept, or removed, depending on whether they carry signal.
• Negations: "not good" means the opposite of "good," and removing "not" as a stop word destroys that. Keep negations, or use n-grams.

Over-cleaning is a classic mistake. Each removal throws away information the model might have used, so let validation scores, not habit, decide.

Bag of Words

Bag of Words represents a document by how many times each vocabulary word appears, ignoring order entirely — like dumping the words into a bag and counting them. Take three tiny documents:

from sklearn.feature_extraction.text import CountVectorizer

docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are pets",
]

vec = CountVectorizer()
X = vec.fit_transform(docs)

print(vec.get_feature_names_out())
print(X.toarray())
print("sparse matrix shape:", X.shape, "| nonzero entries:", X.nnz)

The output shows each document as a row of counts across the vocabulary. "The" appears twice in the first document, so its count is 2, and most entries are zero. Real corpora produce matrices with tens of thousands of columns that are more than 99% zeros, which is why scikit-learn returns a sparse matrix that stores only the nonzero entries. Don't call .toarray() on a real corpus unless you enjoy running out of memory.

Key properties to remember:

• Word order is lost. "Dog bites man" and "man bites dog" look identical.
• The vocabulary is frozen at fit time. Words that appear only in test data are ignored — which is correct behavior, and the reason you must fit the vectorizer on training data only.
• Raw counts overweight common words, which is the problem TF-IDF addresses.
• A binary variant (binary=True) records only presence or absence, which often works well for short texts where repeats carry little meaning.

If you've worked with categorical encoding, the analogy is exact: Bag of Words is one-hot's counting cousin, with one column per vocabulary term instead of one per category.

TF-IDF: Weighting by Distinctiveness

Raw counts treat "the" as the most important word in almost every document, which is useless. TF-IDF (term frequency–inverse document frequency) fixes this by multiplying two quantities:

• Term frequency (TF): how often the word appears in this document.
• Inverse document frequency (IDF): how rare the word is across the whole corpus. Words that appear in nearly every document get a low IDF, and words that appear in few documents get a high one.

The product rewards words that are frequent here but rare elsewhere, which is exactly what makes a word characteristic of a document. In scikit-learn's default (smoothed) form, the inverse document frequency is:

idf(t) = ln((1 + n) / (1 + df(t))) + 1

where n is the number of documents and df(t) is the number containing term t. Each document's TF-IDF vector is then L2-normalized, so every document vector has length one — which stops long documents from dominating simply by containing more words. Compare the two representations:

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

tfidf = TfidfVectorizer()
T = tfidf.fit_transform(docs)

weights = pd.DataFrame(T.toarray(), columns=tfidf.get_feature_names_out()).round(2)
print(weights.to_string())
print("\nlowest idf:", sorted(zip(tfidf.idf_, tfidf.get_feature_names_out()))[:3])

"The," "sat," and "on" (which appear in multiple documents) receive lower weights than words that appear in only one document, such as "mat" or "pets." The idf_ attribute exposes the learned inverse document frequencies, which is useful for sanity checks.

Property	Bag of Words	TF-IDF
Value in each cell	raw count	tf × idf, then L2-normalized
Common words	dominate the vector	down-weighted toward zero
Document length	longer docs score higher	every doc is unit length
Tuning knobs	binary, ngram_range	sublinear_tf, norm, use_idf
Typical role	quick baseline, short texts	default for most text tasks

Sublinear TF scaling (sublinear_tf=True) replaces the raw term count with 1 + log(count), so a word appearing 20 times counts only a few times more than one appearing once, instead of 20 times more. It's a cheap tweak that frequently improves classification, so it's worth trying by default.

The Parameters That Matter

The same options apply to CountVectorizer and TfidfVectorizer:

Parameter	What it does	Typical start
ngram_range	adds bigrams like "not good" and "new york," capturing a bit of word order and negation; grows the vocabulary a lot	(1, 2)
min_df	drops terms appearing in fewer than this many documents (count or fraction) — kills typos and one-off tokens	2–5
max_df	drops terms appearing in more than this fraction of documents — auto-removes corpus-specific stop words	0.5 or 0.9
max_features	keeps only the most frequent terms, capping dimensionality	optional
stop_words	accepts "english" or a custom list	treat as an experiment
lowercase, strip_accents, token_pattern	control normalization and what counts as a token; a custom tokenizer handles special domains	defaults first
analyzer="char_wb" with ngram_range=(2, 5)	character n-grams — robust to typos, handle morphology, work without word boundaries	typo-heavy text
sublinear_tf, norm, use_idf, smooth_idf	adjust the weighting scheme	sublinear_tf=True

A sensible starting configuration is TfidfVectorizer(sublinear_tf=True, min_df=2, max_df=0.5, ngram_range=(1, 2)), then tune from there with cross-validation. The same df cutoffs behave like filter-based feature selection — they remove features before the model ever sees them.

A Complete Classification Example

This example classifies newsgroup posts into four topics, using scikit-learn's 20 newsgroups loader (which downloads the data on first use). The remove argument strips headers, footers, and quoted replies, which otherwise give the model shortcuts like email domains and signatures that don't reflect real text understanding — a textbook case of leakage in text data:

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.pipeline import make_pipeline

cats = ["sci.space", "rec.sport.hockey", "talk.politics.guns", "comp.graphics"]
strip = ("headers", "footers", "quotes")
train = fetch_20newsgroups(subset="train", categories=cats, remove=strip)
test = fetch_20newsgroups(subset="test", categories=cats, remove=strip)

pipe = make_pipeline(
    TfidfVectorizer(sublinear_tf=True, min_df=2, max_df=0.5,
                    ngram_range=(1, 2), stop_words="english"),
    LogisticRegression(max_iter=1000),
)
pipe.fit(train.data, train.target)

print(classification_report(test.target, pipe.predict(test.data),
                            target_names=test.target_names))

Because the vectorizer lives inside the pipeline, its vocabulary and IDF values are learned from the training documents only and applied unchanged to test data — the same leakage rule the preprocessing guide applies to every learned transformer.

Look at What the Model Learned

One of the best features of linear models on TF-IDF is that you can read their reasoning:

import numpy as np

vec = pipe.named_steps["tfidfvectorizer"]
clf = pipe.named_steps["logisticregression"]
names = np.array(vec.get_feature_names_out())

for i, label in enumerate(train.target_names):
    top = np.argsort(clf.coef_[i])[-8:][::-1]
    print(f"{label:<20}", ", ".join(names[top]))

If the top terms for a class are meaningful words, good. If they're artifacts like names, boilerplate, or formatting tokens, the model found a shortcut — clean the data or restrict the features.

Tune the Pipeline

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(
    pipe,
    {"tfidfvectorizer__ngram_range": [(1, 1), (1, 2)],
     "tfidfvectorizer__min_df": [2, 5],
     "logisticregression__C": [0.3, 1, 3, 10]},
    cv=5, scoring="f1_macro", n_jobs=-1,
)
grid.fit(train.data, train.target)
print(grid.best_params_, round(grid.best_score_, 3))

Tune the vectorizer and the classifier together, since the best regularization depends on how many features you give the model. For linear models on text, LinearSVC and SGDClassifier are fast alternatives to logistic regression, and ComplementNB is a strong cheap baseline for imbalanced text.

TF-IDF for Search and Similarity

Because TF-IDF rows are L2-normalized, the dot product between two document vectors is their cosine similarity. That makes a tiny search engine only a few lines:

from sklearn.metrics.pairwise import linear_kernel

corpus = [
    "cheap flights from london to tokyo",
    "best ramen restaurants in tokyo",
    "how to train a neural network",
    "budget airline tickets to japan",
]
vec = TfidfVectorizer(stop_words="english")
M = vec.fit_transform(corpus)

query = vec.transform(["cheap flights to tokyo"])
scores = linear_kernel(query, M).ravel()
for idx in scores.argsort()[::-1][:3]:
    print(round(scores[idx], 3), corpus[idx])

Notice the limitation: "budget airline tickets to japan" means nearly the same thing as the query but shares no vocabulary with it, so it scores poorly. Lexical methods can't see that "cheap" and "budget" are related — a problem called vocabulary mismatch. For retrieval, BM25 is a closely related and usually stronger lexical scoring function, and modern search stacks often combine lexical scoring with embeddings in hybrid retrieval.

Scaling to Large Corpora

• Sparse matrices keep memory manageable, and most scikit-learn linear models accept them directly.
• min_df, max_df, and max_features control vocabulary size.
• HashingVectorizer maps tokens to a fixed number of columns with a hash function, so it needs no vocabulary and works on streaming data and huge corpora. The cost: you can't map columns back to words, and different words can collide. Pair it with TfidfTransformer if you want IDF weighting, and with SGDClassifier.partial_fit for out-of-core training.
• TruncatedSVD on a TF-IDF matrix produces dense, low-dimensional vectors (latent semantic analysis) that capture some synonymy — the text-world cousin of PCA, useful for clustering and visualization.

Where Bag of Words and TF-IDF Fall Short

These methods have real limits:

• No word order or context beyond n-grams, so "not bad" and "bad, not good" are hard to separate cleanly.
• No semantics: synonyms are unrelated columns, and a word with several meanings has one column.
• High-dimensional sparsity, which strains some models and needs enough data.
• Vocabulary drift: new slang, products, and names appear after training and are ignored.
• Sensitivity to tokenization and language, especially for languages without spaces or with rich morphology.
• Weak on very short text, where there are few words to count.

When these bite, move to dense embeddings from a pretrained sentence-embedding model and train a simple classifier on them, or fine-tune a transformer. Build the TF-IDF plus linear model first, because it's fast, cheap, interpretable, and often surprisingly competitive — particularly with domain-specific vocabulary and modest data. Then try embeddings and compare on your own validation set. The two also combine well: concatenating TF-IDF features with embedding features sometimes beats either alone.

Beyond Words: Other Text Features

Counts and weights aren't the only useful signals. Add simple meta-features such as document length, punctuation and capitalization counts, the presence of URLs, digit ratios, and readability scores, and combine them with the TF-IDF matrix using a ColumnTransformer. For some tasks, such as spam or fraud detection, these hand-built features carry real signal.

Practical Tips

• Start with the baseline: TfidfVectorizer(sublinear_tf=True, min_df=2, ngram_range=(1, 2)) plus logistic regression or a linear SVM.
• Keep everything in a pipeline so vocabulary and IDF come from training data only.
• Inspect the top features per class to catch shortcuts and leakage.
• Try character n-grams for typo-heavy, multilingual, or name-heavy text.
• Use max_df or custom stop words to remove domain-specific filler.
• Check class imbalance and use appropriate metrics.
• Read error examples, since misclassified documents show exactly what the features miss.
• Monitor out-of-vocabulary rate in production — a rising share of unseen words signals drift.
• Save the whole fitted pipeline with joblib so serving uses identical preprocessing.

Common Pitfalls

• Fitting the vectorizer on the full dataset before splitting — it leaks vocabulary and IDF statistics.
• Leaving metadata, signatures, or source identifiers in the text — the model learns shortcuts instead of words.
• Removing stop words that include negations — "not good" becomes "good."
• Turning a sparse matrix into a dense array on a large corpus — enjoy your MemoryError.
• Using a huge ngram_range without min_df or max_features — the feature count explodes.
• Comparing models without tuning the vectorizer parameters — you're tuning half the pipeline.
• Assuming TF-IDF captures meaning. It captures vocabulary overlap.
• Dropping the TF-IDF baseline before checking whether a neural model actually beats it.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the difference between Bag of Words and TF-IDF?

Bag of Words puts the raw count of each vocabulary word in its column, ignoring order entirely — "dog bites man" and "man bites dog" are identical vectors. TF-IDF multiplies that count by how rare the word is across the corpus (term frequency × inverse document frequency), so words like "the" that appear everywhere get near-zero weight while distinctive words like "pets" get boosted, and each vector is L2-normalized to unit length. Use counts with binary=True for short texts where repeats don't matter, and TF-IDF with sublinear_tf=True as the default for almost everything else.

Which TfidfVectorizer parameters matter most?

Four do most of the work: sublinear_tf=True replaces raw counts with 1 + log(count) so a word appearing 20 times doesn't count 20× more than one appearing once; min_df (2–5) drops typos and one-off tokens; max_df (0.5–0.9) auto-removes corpus-specific filler words; and ngram_range=(1, 2) adds bigrams like "not good" and "new york" to recover a bit of word order. A strong starting point is TfidfVectorizer(sublinear_tf=True, min_df=2, max_df=0.5, ngram_range=(1, 2)) tuned together with the classifier via GridSearchCV.

How do you prevent data leakage with TF-IDF?

Fit the vectorizer on training documents only — the vocabulary and IDF statistics are learned parameters, exactly like scaler means. Keep it inside a Pipeline so fit_transform runs on train and transform runs on test, never touch the full dataset before splitting, and strip metadata shortcuts from raw text (in the 20 newsgroups example, remove=("headers", "footers", "quotes") blocks email domains and signatures the model would otherwise learn instead of the words).

Is TF-IDF still worth using in the age of embeddings?

Yes — it's the baseline every fancier method should beat, and it often comes closer than you expect: fast, cheap, interpretable, and surprisingly competitive with domain-specific vocabulary and modest data. Build the TF-IDF + linear model first, then try embeddings and compare on your own validation set. The two also combine well: concatenating TF-IDF features with embedding features sometimes beats either alone, and hybrid retrieval that mixes lexical scoring with embeddings is the standard pattern in modern search stacks.

Wrapping This Up

Bag of Words turns documents into word-count vectors, and TF-IDF improves on it by down-weighting words that appear everywhere and boosting words that distinguish one document from the rest. In scikit-learn, CountVectorizer and TfidfVectorizer do the work, with ngram_range, min_df, max_df, and sublinear_tf as the settings that matter most — and a pipeline keeps the learning honest. They're fast, interpretable, and a dependable baseline, with clear limits around word order and meaning.

Is TF-IDF still worth learning in the age of embeddings? Yes, because it's the baseline every fancier method should beat, and it often comes closer than you'd expect. Run the newsgroups example, read the top features per class, then swap in analyzer="char_wb" and see what changes.

Related Articles

• Encoding Categorical Variables: One-Hot, Label, and Target Encoding
• Data Preprocessing for Machine Learning: Complete Step-by-Step Guide
• Feature Selection Methods: Filter, Wrapper, and Embedded Explained
• PCA Tutorial: Dimensionality Reduction with Python