Interpretability vs Explainability in Machine Learning: What's the Difference?

Contents

Figure 1: Interpretability vs Explainability in Machine Learning: What's the Difference? — one model you can read, one model that needs a translator, and the tests that tell you which you have

Ask five machine-learning researchers to define these two words and you will get six answers. Some use them as synonyms, some fight about it, and the argument is older than the current wave of AI tools.

So here is the useful version: what the terms usually mean, why the difference actually matters for the work, and what happened when I tested both approaches on real models.

Interpretability is about understanding the model itself. Explainability is about a tool describing the model to you. Almost every practical disagreement in this field comes from confusing those two positions.

There Is No Official Definition

Start with an honest admission. Lipton's well-known paper says the task of interpretation appears underspecified, and that papers offer myriad notions of what attributes render models interpretable. Doshi-Velez and Kim agree that there is very little consensus on what interpretable machine learning is and how it should be measured.

So any split I give you is a convention, not a law, and the one I use tracks the model-based versus post-hoc categorisation in Murdoch and colleagues' paper:

• Interpretability means the model itself is understandable. You can read it, trace a prediction by hand, and see how it works. Think of a short decision tree or a sparse linear model.
• Explainability means a second tool describes a model after the fact. The model stays opaque, and a method such as feature importance or a surrogate tells you something about its behaviour.

Lipton calls transparency to humans and post-hoc explanations competing notions. That framing is the useful part, because the two approaches pull in different directions rather than simply describing the same thing at different strengths.

Why the Difference Matters

Cynthia Rudin's argument draws the line sharply. Her abstract says that trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practices, and that there is a way forward: to design models that are inherently interpretable. She focuses on high-stakes decisions. Her abstract does not make a claim about accuracy trade-offs, so I will not put one in her mouth.

Murdoch and colleagues offer a way to evaluate either approach. Their PDR framework has three desiderata:

1. Predictive accuracy — does the model predict well?
2. Descriptive accuracy — does the description match what the model actually does?
3. Relevancy — is the description useful to the human audience reading it?

A model or an explanation can score well on one axis and badly on another, which is exactly how a good explanation can still mislead. To see what the practical difference looks like, I ran real scikit-learn models. Everything below uses small datasets, so treat it as an illustration rather than a law.

Is There an Accuracy Cost for Readable Models?

I compared models you can read against black boxes, using 5-fold cross-validation repeated three times:

Model	Breast cancer (accuracy)	Wine (accuracy)
Logistic regression	0.975	0.983
Logistic regression, L1 (sparse)	0.969	0.981
Decision tree, depth 3	0.923	0.918
Decision tree, depth 5	0.931	0.912
Random forest	0.960	0.978
Gradient boosting	0.960	0.947

On both real datasets, plain logistic regression matched or beat the black boxes, and the classic "interpretable" option, the decision tree, did worst. That is a useful surprise for anyone who assumes readable means weaker.

Fold-to-fold spread was 1 to 4 points, and I ran no formal significance test, so treat gaps of a point or two as ties. Then I tried a problem built to defeat simple models, a nonlinear regression benchmark:

Model	R²
Linear regression	0.722
Decision tree, depth 3	0.543
Decision tree, depth 5	0.607
Gradient boosting	0.890

Here the black box wins by a wide margin, so the pattern flips on data that needs nonlinearity. The honest lesson is that the cost of interpretability depends on the data rather than on the method. One caveat on my test: I did not try interpretable nonlinear models such as generalized additive models, which might close part of that gap.

How Well Do Post-Hoc Explanations Copy the Model?

A global surrogate is a small tree trained to mimic a black box, and its fidelity is how often it agrees with the model it explains. I fitted surrogates to a random forest that scored 0.953 on held-out data:

Surrogate depth	Agrees with forest	Accuracy on true labels
1	0.912	0.889
2	0.942	0.906
3	0.924	0.901
4	0.942	0.906
6	0.930	0.906

Fidelity wobbles by depth, and there are only 171 test rows, so a single row moves it by 0.6 points. Read the pattern, not the digits. The surrogate agrees with the forest roughly 92% to 94% of the time, which leaves about 6% disagreement, larger than the forest's own 4.7% error rate. On the nonlinear problem, surrogate trees reproduced the boosted model only partially, with fidelity R² of 0.53, 0.70 and 0.78 at depths 2, 3 and 5.

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# a surrogate is just a small model fitted to the black box's predictions
surrogate = DecisionTreeClassifier(max_depth=2, random_state=0)
surrogate.fit(X_train, forest.predict(X_train))
fidelity = accuracy_score(forest.predict(X_test), surrogate.predict(X_test))
print("fidelity:", fidelity, "| forest accuracy:", accuracy_score(y_test, forest.predict(X_test)))

Fidelity also says nothing about correctness, because a surrogate can copy a mistake perfectly. A model that is wrong for the same reason as its explanation produces a beautiful, faithful, useless report.

Do Different Models Explain the Data Differently?

I trained five accurate models on the breast-cancer data, all between 0.936 and 0.959 test accuracy, and asked each for its top features by permutation importance:

Model	Top feature
Random forest	worst radius
Gradient boosting	worst texture
Logistic regression	radius error
SVM (RBF kernel)	mean fractal dimension

Five accurate models, four different answers to "what matters most," and no obvious winner among them. The rank correlations between the importance vectors were low: the two random forests with different seeds agreed at 0.86, but gradient boosting against the forests came out at 0.18 and 0.28, logistic regression against the forests at 0.44 and 0.49, and the SVM against everything from −0.04 to 0.28.

A plausible reason is that many breast-cancer features are near-duplicates, since radius, perimeter and area measure similar things, so credit can shift between them. I did not test that mechanism. Either way, the lesson holds: an explanation describes the model you fitted, not the data in the abstract. "Explain the data" and "explain the model" are different jobs, and the second is the one your importance chart is doing.

Is an Interpretable Model Always Readable?

I expected trees to turn into spaghetti as they grew. They did not, at least here:

Max depth	Accuracy	Leaves
2	0.915	4
3	0.923	8
5	0.931	18
8	0.926	22
Unlimited	0.927	22

Even the unrestricted tree has only 22 leaves, and accuracy stops improving past depth 5, so on a 569-row dataset size alone did not make the tree unreadable. A bigger problem, with thousands of features, would behave differently.

Lipton also warns against assuming that linear models are automatically interpretable and deep networks automatically not, saying he will question the oft-made assertions to that effect. A 30-feature regression with correlated inputs can be harder to read than it sounds. I did not measure that, so treat it as his caution rather than my result.

The moment this became personal: my first pass through the surrogate fidelity table treated the jump from 0.912 to 0.942 as a finding about depth, and started a sentence about depth 2 being the sweet spot. With 171 test rows, that gap is about five rows. I now compute the spread before I read the pattern, and the sentence never gets written.

How to Choose

The question is not which word wins. It is which of these situations you are in:

1. High stakes, structured data? Start with an interpretable model and check what a black box actually gains. On my two real datasets, it gained nothing.
2. Complex data where simple models fail? Use the black box, then validate every explanation by checking fidelity, stability across seeds, and agreement across models. If you need the attribution tooling, the Captum tutorial and the SHAP and LIME comparison cover the mechanics, and the XAI tools guide covers the landscape.
3. Need to satisfy an audience? Ask the PDR questions: is it accurate, is the description accurate, and is it relevant to the people reading it?
4. Never report an explanation without its test. A surrogate's fidelity, an importance score's variability and a model-agreement check all take minutes.

Before you present a model or an explanation:
[ ] stated whether the model itself is readable, or a tool is describing it
[ ] compared a simple model against the black box on the same split
[ ] reported the surrogate's fidelity, not just its shape
[ ] checked importance stability across seeds and across model families
[ ] answered the PDR questions for the audience you are presenting to
[ ] included the caveat that the explanation describes this fitted model

Common Pitfalls

1. Treating the two words as interchangeable. The distinction decides whether you can inspect the model or only a description of it.
2. Assuming interpretable means less accurate. On my structured datasets logistic regression beat the forests, and the decision tree lost to both.
3. Assuming a black box always wins. It won on the nonlinear benchmark and lost on the tabular classification ones, so measure rather than assume.
4. Reading one fidelity number as a conclusion. With a small test set, a single row moves the metric.
5. Mistaking fidelity for correctness. An explanation can reproduce a model's mistakes perfectly.
6. Presenting one model's importance ranking as a property of the data. Five accurate models named four different top features.
7. Skipping the audience test. PDR's relevancy axis is the one most reports ignore, and it is the one the reader notices.

Check the edition date before you buy. The frameworks here have been stable for years, but the code samples have not.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the difference between interpretability and explainability?

Interpretability usually means the model itself is understandable, so you can read it and trace a prediction by hand, as you can with a short decision tree or a sparse linear model. Explainability usually means a second tool describes a model after the fact, leaving the model opaque while something like feature importance or a surrogate reports on its behaviour. Lipton calls these transparency to humans and post-hoc explanation competing notions, which is a useful way to see them, because the two approaches pull in different directions. Neither term has an official definition, so what matters is which model you can inspect yourself.

Do interpretable models lose accuracy?

Not necessarily, and on small structured data they may lose nothing at all. In my tests plain logistic regression reached 0.975 accuracy on breast cancer and 0.983 on wine, matching or beating random forests and gradient boosting on both datasets. The classic interpretable option, a decision tree, was the weakest of the group at depths three and five. But on a nonlinear regression benchmark the black box won by a wide margin, scoring an R squared of 0.890 against 0.607 for a depth-five tree. The cost of interpretability depends on the data rather than on the method.

What is surrogate fidelity?

Fidelity is how often a surrogate model, typically a small tree trained to mimic a black box, agrees with the model it is explaining. In my test, surrogates fitted to a random forest with 0.953 held-out accuracy agreed with it roughly 92% to 94% of the time, depending on depth. That leaves about 6% disagreement, which is larger than the forest's own 4.7% error rate, so the explanation is less faithful than the model is accurate. Fidelity also says nothing about correctness, because a surrogate can copy a mistake perfectly.

Why do different models rank features differently?

Because an explanation describes a particular fitted model rather than the data in the abstract. I trained five accurate models on the same breast-cancer data and they named four different top features by permutation importance. Rank correlations between their importance vectors were low, with the two random forests agreeing at 0.86 while gradient boosting against the forests came out at 0.18 and 0.28. A likely reason is that many of those features are near-duplicates, such as radius, perimeter and area, so credit can shift between them. I did not test that mechanism, but the practical lesson holds.

What is the PDR framework?

PDR stands for predictive accuracy, descriptive accuracy and relevancy, and it comes from Murdoch and colleagues as a way to evaluate a model or an explanation. Predictive accuracy is whether the model predicts well, descriptive accuracy is whether the description matches what the model actually does, and relevancy is whether the description is useful to the people reading it. The value of the framework is that a model or explanation can score well on one axis and badly on another, which is how a genuinely good explanation can still mislead an audience.

Should I use an interpretable model or a black box?

Start with the question you are answering. On high-stakes decisions with structured data, fit an interpretable model first and check what a black box actually gains, because in my two real datasets it gained nothing. On complex data where simple models clearly fail, use the black box, then validate every explanation by checking fidelity, stability across seeds and agreement across models. Whichever route you take, report the test alongside the explanation, because the tests take minutes and the trust they earn lasts much longer.

Wrapping Up

Interpretability asks whether you can understand the model. Explainability asks whether a tool can describe it. Neither word has an official definition, and the two approaches trade off in ways that depend on your data and your stakes rather than on a preference for one camp.

My tests suggested three things: simple models often cost nothing on small structured data, post-hoc explanations copy their models imperfectly, and different accurate models tell different stories about the same data. Before you trust any explanation, test it — and ask which model it describes.

Which situation are you in right now: a readable model you have not benchmarked, or a black box whose explanation you have not stress-tested?

Related Articles

• Explainable AI for LLMs: How to Interpret LLM Outputs
• SHAP vs LIME: Which Explainability Method Should You Use?
• Permutation Feature Importance Explained with Scikit-learn
• Best Explainable AI and Model Interpretability Tools
• InterpretML: Explainable Boosting Machines