Reading Inside the Model: What the 'Already Clean' Essay Means for AI Testing
Penhale Mutual's model-risk committee had one item left on the agenda: the claims-triage assistant's annual review. The vendor's account manager had sent a note the night before. New research, it said, shows that around 90% of the neurons in a modern language model can be read. The vendor would soon offer an "inside-the-model audit" that could show the assistant never uses a claimant's postcode when it scores fraud risk. Could the committee retire the external fairness test suite, which cost a week of the QA team's time every quarter?
The chair turned to the QA lead and asked whether this changed the test plan.
(Penhale Mutual is a composite insurer we use across ShiftQuality's AI-quality series, built from patterns we've seen. It isn't a real company, and its details are illustrative.)
The research the vendor meant is real. On 15 October 2026, Nick Cammarata, an interpretability researcher at Goodfire and formerly at OpenAI, published a long essay called Neural Networks Are Already Clean. It argues that the inside of a neural network is far more readable than the field has believed for most of a decade. It drew unusually strong reactions, enthusiastic and sceptical. It also ended up in vendor emails within days, which is why it matters to people who test and assure AI, not just to researchers.
This article covers what the essay claims, which parts are evidence and which are hypothesis, and what the QA lead should say to the committee.
The old picture said the inside of a model was a junk drawer
A neural network is a very large set of numbers that nobody wrote by hand. Training adjusts them until the model predicts well. The units inside it, neurons, each respond more or less strongly to what passes through. Interpretability is the field that tries to say what each part is doing.
Early on, researchers looked at individual neurons in an image model called InceptionV1 and found many that seemed to respond to unrelated things. The most famous one seemed to fire on cat faces and on the fronts of cars. A neuron like that is called polysemantic, meaning it has many meanings.
The explanation the field settled on was superposition. A model has far more concepts to represent than it has neurons. So, the theory goes, it packs unrelated concepts into the same neurons, choosing pairs that rarely appear together, so they don't interfere. Cats and cars share a drawer because there was room, not because they're related.
If that's true, looking at a neuron tells you little. So the field built a second layer of tooling, such as sparse autoencoders, which are trained to unpack a model's activity into cleaner "features". Interpretability became a decoding problem. By 2023 the measured numbers looked bleak. In Anthropic's Towards Monosemanticity work, the median neuron in a small one-layer model scored 0 on a 14-point rubric, meaning the rater couldn't form any hypothesis about it. Cammarata reports that the automated-interpretability project he worked on at OpenAI that year found about 1.7% of neurons interpretable by its most generous measure.
The essay's claim: the mess was mostly in the instruments
Cammarata's argument is that superposition was treated as the default explanation when it should have been the last one. He went back to the original evidence with better tools and found ordinary explanations.
The cat-and-car neuron is a V-shape detector. A cat's face, with pointed ears and a narrow chin, and the front of a car both make a V. The neuron also responds to V-shaped flames, fox faces and water jets. The check that persuades is its opposite pole: what pushes the neuron the other way is a round shape, an O, in the same place. A random cat-and-car mixture would have no reason to have a circle as its opposite. A V detector would.
The famous superposition circuit was about pose. In the 2020 Zoom In paper, which Cammarata co-wrote, a car feature appeared to be hidden among dog neurons. He now reads it as the dogs and cars sharing a pose: facing the camera. He says no second circuit of that shape has been published.
Neurons grow outwards, like branches. Cammarata retrained the image model from scratch and watched single neurons over training. One started on yellow rapeseed fields. It then picked up tennis balls, dogs sitting in grass and, by the end, broccoli. Viewed only at the end, it looks arbitrary. Viewed step by step, each addition is related to the one before. He calls this recursive generalization. A language example is a neuron that starts on the letters "iy", extends to "diy" and ends on "do it yourself", which no longer contains "iy".
The instruments added mess of their own. Three findings stand out for anyone who measures software:
The visualization tool took a shortcut. Feature visualization generates the image that most excites a neuron. In deep layers it found that drawing dog features excited almost anything, so many neurons looked secretly about dogs. The essay calls this reward hacking.
Explanations were graded on prediction, not truth. The 2023 automated method scored an explanation on how well it could simulate when a neuron fires, not on whether it was correct. The essay notes that these two often come apart: "cats and cars" could do well on many tests of the V neuron while missing what it really is.
The test data was too narrow. The team first gave its agents two paragraphs of context. That made every neuron that depends on long context look polysemantic.
Architecture mattered. A modern ConvNeXt model of the same size as InceptionV1, trained on the same data, had neurons that mostly stayed on their original idea instead of sprawling. In its last layer, Cammarata reports about 99% of neurons are easily interpretable. That figure is for ConvNeXt; for InceptionV1 his numbers rise from about 5% with early tools to about 70% with better tools and retraining.
Then the language models. In what he calls preliminary audits of Qwen3-32B, an open-weight model, his team estimates that roughly 90% of neuron poles have one main job, about 7% have two related jobs and about 2–4% aren't understood yet. Much of that labelling was done by AI agents acting as automated scientists: proposing explanations, designing tests and running them at a scale people couldn't.
Some of the examples are memorable. A neuron that seemed to fire on "pr", Base64 strings and Bengali text turned out to be a "pr" neuron throughout: decode the strings and transliterate the Bengali, and they contain "pr". A neuron labelled a "polysemantic comma neuron" tracked Colorado. The model uses commas to store a running summary, and steering the neuron up made the model plan a hike.
How much of this is evidence?
Not all of it carries the same weight, and the essay is careful to say so. It's labelled "an informal research essay and working hypothesis". The companion technical paper isn't out yet. The table below separates what has been shown from what is being proposed.
Claim | Evidence offered | Status |
The cat-and-car neuron is a V detector | Example images, plus the round shape that drives it the other way | Shown for one famous neuron, and anyone can inspect it |
The Zoom In superposition circuit is about pose | Re-analysis with modern feature visualization | A reinterpretation of one circuit |
Neurons grow by recursive generalization | Training checkpoints for a handful of neurons | Shown in detail for a few cases; the claim that it holds generally is hypothesis |
About 99% of the last layer is readable | A retrained ConvNeXt, not InceptionV1 | One model, the author's own rating |
About 90% of Qwen3-32B neuron poles have one main job | Preliminary audit, largely done by agents | Depends on thresholds; the paper is still to come |
Similar results in trillion-parameter models such as Kimi K3 | One attribution graph shown; "full audits" not yet started | Early |
Interpretability may be far more tractable than assumed | Argument from the above | The essay itself marks this step "highly speculative" |
The 90% figure carries the most weight in the headlines, and the essay attaches its own warning to it. The share changes with where you set the threshold for counting weak activations, which layer you look at, and what you count as "one thing". The author calls the single "cleanliness" score "a broad pointer", not a precise target.
Researchers outside Goodfire are split. Writing up the reaction in The Decoder, Maximilian Schreiner reports that Google DeepMind's Neel Nanda found the work interesting but said it doesn't change the big picture. Nanda still thinks superposition is real, because models store far more facts than they have neurons. He doubts that findings from small image models transfer to language models, and he thinks the essay defines monosemanticity loosely. Others called it a landmark. Both camps agree on the next step: someone else has to reproduce it.
The essay names its own biggest risk, too. If you go looking for a sensible story behind every neuron, you may find one whether or not it's there. Cammarata's answer is that each of his alternative causes can be tested. A better tool either resolves the neuron or it doesn't. Retraining shows whether the architecture was the problem. Steering a suspected Colorado neuron either produces Colorado talk or it doesn't. That is the right answer. It's also a demand for evidence that a reader can make of every claim in the essay, and of every vendor who cites it.
The lesson for testers is about instruments, not neurons
Read as a testing story, the essay is about a field that blamed the system under test for what were partly faults in its own measurement. Every QA lead has seen the same pattern: a suite that reports chaos because the harness is flaky, the oracle is wrong or the data is too narrow.
The three instrument faults in the essay map directly onto AI evaluation work you may be running today.
What went wrong in interpretability | The same fault in AI testing | A check you can run |
Feature visualization found an easy shortcut (dogs) that excited almost everything | An LLM judge or scorer that rewards a surface feature, such as length, confident tone or a keyword | Feed the grader deliberately wrong answers that have the surface feature. If they pass, the grader is measuring the feature |
Explanations were scored on prediction, not truth | An eval that passes when the output matches a pattern, not when it is correct | Take 20 graded outputs, have a person check correctness, and count how often the grader's pass and the human's verdict disagree |
Two paragraphs of context made long-context behaviour look random | Eval sets built from short, clean prompts that never exercise the context real users send | Sample real inputs by length. If your eval set has no examples above your median production input length, it can't see that behaviour |
None of this needs access to model weights. It's the same discipline the essay applies to neurons, applied to your own test rig. The useful question is the one Cammarata asked himself: is the system messy, or am I looking at it badly?
What it does and doesn't change for assurance today
Back at Penhale, the QA lead had a choice: accept an inside-the-model audit in place of the fairness suite, or add it alongside. Walking that one choice through shows where the claim actually helps.
Context. The triage assistant runs on a hosted model behind the vendor's API. Penhale has no access to its weights. The fairness suite sends matched pairs of claims that differ only in postcode, and checks that the fraud score doesn't move by more than an agreed tolerance.
The decision. Retiring the suite would trade a behavioural test, which checks what the assistant actually does with Penhale's claims, for a claim about the inside of a model Penhale can't inspect.
The reasoning. The QA lead asked the vendor three questions. They work for any interpretability claim.
Which model, and at what threshold? The 90% figure is for Qwen3-32B neuron poles, from a preliminary audit, and moves with the threshold. The vendor's model is a different model. Any number the vendor quotes should come with its own model name, layer coverage and the rule for what counts as "one job".
Was the explanation tested by intervention, or matched to examples? A label such as "this neuron tracks postcode" is a hypothesis until someone removes or steers the neuron and the behaviour changes as predicted, with a random control beside it. The essay does this itself. When strongest activations were removed from a Qwen model, it stopped reciting Psalm 23 and claimed the psalm didn't exist. Random interventions matched for count or size left the verse intact. That control is what makes the result evidence.
Who checked the explainer? If agents wrote the labels, what was their error rate against human review, and how were they scored? The 2023 lesson is that an explainer can be graded well and still be wrong.
The result. The vendor could answer the first question in general terms only. It had no intervention evidence for postcode specifically, and it had no external review of its labels yet. The committee kept the fairness suite. It added the three questions to the vendor's annual due diligence, so that a future audit would arrive with evidence attached.
The lesson. Interpretability is a new instrument, and a potentially powerful one. Instruments add to behavioural testing; they don't replace it. Even a perfectly readable model only tells you what its parts do. It doesn't tell you whether the feature meets Penhale's fairness requirement, because that requirement lives in Penhale's scope, not in the model. You can't test your way to a requirement you never wrote down, and you can't read your way to one either.
There is a real way the essay could matter more soon. Anthropic showed in 2024 what reading and steering can look like. It turned up one internal feature linked to the Golden Gate Bridge, and its Claude model started mentioning the bridge in most replies, relevant or not. If finding and steering specific mechanisms becomes routine, it opens tests that behavioural suites can't run. You could check whether a model represents a sensitive attribute at all, or watch a mechanism linked to evasive answers during a release candidate's evaluation. That is the case for following this work closely. It isn't yet a case for changing a test plan.
What to do this week
Audit one grader. Take 20 outputs your eval suite has scored. Have a person judge correctness without seeing the grader's verdict, then count disagreements. If more than a couple disagree, your grader is measuring something other than correctness, and fixing that is worth more than any interpretability result this year.
Check your eval data against production. Compare the length and shape of eval prompts with real ones. Add long-context cases if you have none.
Add the three questions to vendor due diligence. Ask for model, threshold, intervention evidence and explainer error rate whenever a supplier cites interpretability.
Put a review date in the diary. The companion paper and independent replications will decide how much of the 90% holds. Revisit your stance when they're out, not before.
Final takeaway
Neural Networks Are Already Clean makes a serious, partly evidenced case: much of the apparent chaos inside neural networks came from weak tools, narrow data and older architectures, and modern models may be far more readable than the field assumed. The worked examples are persuasive, the headline numbers are preliminary and threshold-dependent, and the paper that would let others check them hasn't been published. For people who assure AI, the immediate value is the method, not the number. Check the instrument before you blame the system, test explanations by intervention with a control beside them, and keep behavioural tests for the requirements you own. Start by auditing one grader this week.
Sources
Cammarata, N., Neural Networks Are Already Clean, Goodfire, October 2026. An informal research essay and working hypothesis; Qwen3-32B preliminary audit (about 90% / 7% / 2–4%) and its threshold caveat; ConvNeXt results; recursive generalization; the 2023 baselines as reported by the author; the Psalm 23 intervention with random controls. Language-neuron research with Curt Tigges. https://already-clean.goodfire.com/
Schreiner, M., "Is AI interpretability close to a breakthrough? A new essay has researchers divided", The Decoder, 16 October 2026. Reactions, including Neel Nanda's criticism. https://the-decoder.com/is-ai-interpretability-close-to-a-breakthrough-a-new-essay-has-researchers-divided/
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., "Zoom In: An Introduction to Circuits", Distill, 2020. https://distill.pub/2020/circuits/zoom-in/
Olah, C., Mordvintsev, A. and Schubert, L., "Feature Visualization", Distill, 2017. https://distill.pub/2017/feature-visualization/
Bills, S., Cammarata, N., Mossing, D., et al., "Language Models Can Explain Neurons in Language Models", OpenAI, 2023. Explanations scored by simulating activations. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html
Bricken, T., Templeton, A., Batson, J., et al., "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning", Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html
Anthropic, "Golden Gate Claude", May 2024. https://www.anthropic.com/news/golden-gate-claude
Related reading
Keep learning. This article is part of the Shipping AI guide: defining done, evaluating output you can't judge by eye, and running AI once real users depend on it.
