Explainable AI for LLMs: How to Interpret LLM Outputs

Contents

Figure 1: Explainable AI for LLMs: How to Interpret LLM Outputs — four kinds of explanation, four different questions, and the stress test that decides which one to trust

Your chatbot gives a confident, fluent answer. You ask why, and it gives a confident, fluent reason. Should you believe the reason? That question sits at the centre of LLM interpretability, and it does not have a comfortable answer.

There are four common ways to get an explanation out of a language model. Each answers a different question, and each can mislead you in its own way. I walk through all four, test one on a toy model where I know the truth, and finish with a checklist.

Every LLM explanation method answers a narrower question than it appears to, so the useful skill is not picking the best method, it is knowing which question you just asked.

The Four Kinds of Explanation

Approach	The question it answers	The main risk
Attention weights	Where did the model look?	Looking is not the same as mattering
Token attribution	Which words changed the output?	The answer depends on the metric and the perturbation
Ask the model	What reason does the model give?	Plausible stories that omit the real cause
Look inside	What does the model represent internally?	Findings are preliminary and rarely tested on your task

1. Look at attention. Where did the model look?
2. Attribute to input tokens. Which words changed the output?
3. Ask the model. Have it write out its reasoning.
4. Look inside. Decode hidden states or find internal features.

1. Attention: Where the Model Looks

Attention weights are sitting right there in the model, which is exactly why everyone wants to read them. The research is split on whether you should.

Jain and Wallace tested two properties. They found that learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and that one can identify very different attention distributions that nonetheless yield equivalent predictions. Their conclusion is blunt: standard attention modules do not provide meaningful explanations and should not be treated as though they do.

Wiegreffe and Pinter pushed back. They argue the claim depends on how you define explanation, and they offer four alternative tests, including a uniform-weights baseline and an adversarial attention training protocol. Their verdict is that prior work does not disprove the usefulness of attention mechanisms for explainability. I read only these two abstracts, so the detail is theirs.

So the debate is unsettled, and the honest position is that neither side has finished the argument. I built a toy to see what the arguments look like in numbers. It is a one-head attention classifier in NumPy with random weights, not a trained LLM, and everything below shows what is possible inside that toy.

A constructed example

I hand-built a six-token sentence where two filler tokens attract attention but carry no sentiment:

Token	Attention	Leave-one-out effect	Gradient × input
the	0.435	+0.074	−0.125
movie	0.032	−0.007	+0.012
was	0.032	+0.003	−0.001
not	0.072	+0.193	−0.388
great	0.072	−0.272	+0.561
.	0.356	+0.053	−0.096

The token with the most attention is "the." The tokens with the biggest influence are "great" and "not," which carry about six times less attention each. Reading the attention column alone would have pointed you at a determiner and missed the sentiment entirely.

Two details matter:

• "the" is not inert. Deleting it moves the output by 0.074, because it frees attention for the other tokens, so a low-influence token can still be doing real work.
• Leave-one-out and gradient × input even disagree in sign for "the." They answer different questions: delete this token, versus nudge this embedding.

How often do attention and influence agree?

Across 2,000 random toy models with 10 tokens each, attention did better than I expected:

Attention sharpness	Rank agreement with leave-one-out	Top-attention token is also top-influence token
1.0	0.648	53% (chance: 10%)
3.0	0.915	79%

That is a decent proxy by the numbers. But part of that agreement is baked into the toy, because attention scores and token values both come from the same embedding, so big-norm tokens get both. Trained models do not have to share that link. Read the table as "attention is not automatically useless, and not automatically honest", rather than as a measurement of real models.

2. Token Attribution: What Changed the Output

The sturdier family of methods perturbs the input, either by deleting tokens or by measuring gradients. I tested a deletion check: remove the top-k tokens ranked by each method and measure how far the output moves, averaged over 600 random toy models.

Ranking method	k=1	k=2	k=3
Random order	0.201	0.486	0.698
Attention	1.536	1.743	1.913
Gradient × input (absolute)	1.519	1.400	1.538
Leave-one-out	1.588	1.241	1.264

Every informed method crushes random order, which is the minimum bar and worth confirming before trusting anything more complicated. Now the surprise: at k=2 and k=3, attention-ranked deletion moved the output the most, ahead of leave-one-out.

My best guess is that single-token effects cancel when tokens come out together, and that removing high-attention tokens shifts attention mass onto the rest. I did not test that mechanism, and this metric rewards large disturbances, so it does not prove attention is the more faithful signal.

# deletion check: remove the top-k tokens an explanation ranked highest
base = score(model, tokens)
for k in (1, 2, 3):
    keep = [t for t in tokens if t not in top_k(ranking, k)]
    print(k, abs(score(model, keep) - base))

The lesson is the portable one: a faithfulness metric is a choice, and the ranking of methods depends on the choice. If someone shows you a deletion curve, ask what it rewards.

3. Asking the Model to Explain Itself

Chain-of-thought explanations read well. The question is whether they tell the truth about why the model answered the way it did.

Turpin and colleagues tested this by adding biasing features to inputs, such as reordering multiple-choice options so the answer is always the first one. When models were biased toward wrong answers, accuracy fell by as much as 36% across 13 BIG-Bench Hard tasks, tested on GPT-3.5 and Claude 1.0. The explanations did not mention the bias. On a social-bias task, the models justified stereotyped answers rather than flagging them. The authors conclude that chain-of-thought explanations can be plausible yet misleading, which risks increasing our trust in LLMs without guaranteeing their safety.

Lanham and colleagues went further and intervened on the reasoning itself, adding mistakes or paraphrasing it, then watching how predictions changed. They report that as models become larger and more capable, they produce less faithful reasoning on most tasks studied. That direction of travel is the uncomfortable part.

Treat a model's self-explanation as a story to test, not as evidence to trust. I read abstracts only for both papers, so the nuance is in the originals.

4. Looking Inside the Model

Interpretability researchers read hidden states directly. Two directions stand out.

Decoding layers. The logit lens turns a hidden state into a guess about the next token, which lets you watch a prediction form layer by layer. The tuned-lens paper calls it a refinement of the earlier logit lens technique, which yielded useful insights but is often brittle. It trains an affine probe for each block in a frozen pretrained model, and the authors tested it on autoregressive language models with up to 20 billion parameters. If you have already worked with attribution on vision or tabular models in the Captum tutorial, the probe-per-layer idea will feel familiar.

Finding features. Anthropic trained sparse autoencoders on Claude 3 Sonnet activations and reported extracting high-quality features. Their examples include features for famous people, for countries and cities, and for type signatures in code. They also report that clamping a Golden Gate Bridge feature to ten times its maximum activation made the model start identifying as the bridge, and they describe features related to security vulnerabilities, bias, lying and bioweapons-related content.

The authors' own caveats matter more than the headlines. They warn not to read too much into the mere existence of such features, call the work very preliminary, and say it does not show any feature is actually useful for safety. That honesty is the right template for everyone else in this field, and it is worth copying before you cite the result in a slide.

Can a Different Attention Pattern Produce the Same Answer?

One more toy result, and it is the cleanest way to see what the attention debate is about. Because a model's output is a weighted average of per-token scores, many different attention patterns can average to the same number.

In my toy, I found a different attention pattern with the same output, within 0.02, and over half its attention on a different token, in 295 of 300 random models. That is 98%, and it shows the possibility rather than the practice, which is the same point Jain and Wallace make. It does not show that trained models exploit it, and the gap between those two statements is where most interpretability arguments go wrong.

The moment this became personal: I built the six-token example to demonstrate the disagreement, and my first instinct on seeing the attention column was to write a sentence about the model attending to determiners. The influence columns were already on screen and said otherwise. If I had been analysing someone else's model without ground truth, that sentence would have shipped.

Common Pitfalls

1. Using attention alone as your explanation. Pair it with a perturbation test, because looking and mattering are different things.
2. Failing to say which question you asked. "Which token mattered if deleted?" and "which token drove the gradient?" can differ, even in sign, for the same input.
3. Publishing an explanation you never tested. Delete, mask or scramble the tokens an explanation points to, and check that the output moves as claimed.
4. Ignoring what the metric rewards. Faithfulness scores are choices, and a method's ranking can flip when you change yours.
5. Treating self-explanations as evidence. Bias the prompt deliberately and see whether the explanation admits it.
6. Presenting internal-feature findings as settled. Label them preliminary unless you have tested them on your own task.
7. Assuming the toy transfers. My numbers come from a one-head NumPy classifier with random weights, which shows what is possible, not what your model does.

Checklist before you present an LLM explanation:
[ ] the method is named alongside the question it actually answers
[ ] attention is paired with a perturbation test, never shown alone
[ ] the tokens the explanation highlights were deleted or masked, and the output moved
[ ] the faithfulness metric is stated, and you know what it rewards
[ ] self-explanations were tested with a biasing prompt
[ ] internal-feature results are labelled preliminary
[ ] an alternative explanation with the same output was searched for

Check the edition date before you buy. Interpretability findings from a year ago can already be superseded, and the code samples age faster still.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Can I use attention weights to explain an LLM?

Not on their own, because the research is genuinely split on whether attention is an explanation. Jain and Wallace found that learned attention weights are frequently uncorrelated with gradient-based feature importance, and that different attention distributions can yield equivalent predictions. Wiegreffe and Pinter pushed back, arguing the verdict depends on how you define explanation and offering alternative tests. In my toy, the token with the most attention carried far less influence than two other tokens, so attention and influence can disagree. Pair attention with a perturbation test before you present it.

What is the best way to attribute an LLM's output to input tokens?

Perturbation-based attribution, in which you delete or mask tokens and measure how far the output moves. In my toy tests every informed ranking beat random order by a wide margin, with leave-one-out deletion, gradient times input, and attention all clearly ahead of random at k equals one, two and three. Which method wins depends on the metric you pick, because a faithfulness metric is a choice rather than an absolute. Report the metric alongside the ranking, and remember that deleting tokens measures a different question from nudging an embedding.

Are chain-of-thought explanations faithful?

Often not, and the evidence is uncomfortable. Turpin and colleagues added biasing features such as reordering multiple-choice options so the answer was always the first one, and accuracy fell by as much as 36% across 13 BIG-Bench Hard tasks on GPT-3.5 and Claude 1.0, while the explanations never mentioned the bias. Lanham and colleagues intervened on the reasoning itself and reported that as models become larger and more capable, they produce less faithful reasoning on most tasks studied. Treat a self-explanation as a story you can test, not as evidence.

What is the logit lens?

The logit lens is a technique for reading a hidden state directly by turning it into a guess about the next token, which lets you watch a prediction form layer by layer. The tuned-lens paper describes its own method as a refinement of the earlier logit lens technique, which yielded useful insights but is often brittle. It trains an affine probe for each block in a frozen pretrained model, and the authors tested it on autoregressive language models with up to 20 billion parameters. It is a diagnostic, not a proof of what the model knows.

Are interpretability features from sparse autoencoders reliable?

Treat them as preliminary, which is what the researchers themselves say. Anthropic trained sparse autoencoders on Claude 3 Sonnet activations and reported extracting high-quality features, including ones for famous people, countries and cities, and type signatures in code. They also reported that clamping a Golden Gate Bridge feature to ten times its maximum activation made the model identify as the bridge. Their own caveats matter most: they warn against reading too much into the mere existence of such features, call the work very preliminary, and say it does not show any feature is useful for safety.

How do I test whether an LLM explanation is trustworthy?

Stress-test it rather than admiring it. Delete, mask or scramble the tokens the explanation points to and check whether the output moves as predicted. Change the metric and see whether the method ranking survives. Bias the prompt deliberately and check whether a self-explanation admits the bias or invents a story around it. Try an alternative explanation that produces the same output, which in my toy was possible in 98% of random models. If several independent tests agree, the explanation is worth showing.

Wrapping Up

Attention shows where the model looked, attribution shows what changed the output, self-explanation shows what the model says, and internal features show what the model represents. None of them alone shows why. Each answers a narrower question than the label suggests, and the optimism comes from forgetting that gap.

My view is that the best habit here is to treat every explanation as a claim you can test: delete the token, bias the prompt, change the metric, and see whether the story survives. An explanation that survives three different tests is worth showing. One that survives none is worth knowing about before someone else finds out.

Which explanation will you stress-test first: the attention map, the chain-of-thought, or the internal feature?

Related Articles

• SHAP vs LIME: Which Explainability Method Should You Use?
• Captum Tutorial: Integrated Gradients for PyTorch Models
• Permutation Feature Importance Explained with Scikit-learn
• Best Explainable AI and Model Interpretability Tools
• MMLU Explained: What the Benchmark Does and Where It Falls Short