The "black box" framing is becoming obsolete. Mechanistic interpretability, the practice of reverse-engineering a neural network's inner workings, has reached a point where researchers can watch a model think in real time, at the level of human-readable concepts.
The landmark piece driving this conversation is Anthropic's On the Biology of a Large Language Model (2025). Jay Hack's summary is a useful tour through its core findings.
Why individual neurons are not enough
The fundamental challenge is superposition. A single neuron participates in many unrelated concepts, and any given concept is smeared across many neurons. You cannot read meaning off one unit. That makes naive activation inspection nearly useless.
How circuit tracing works
Anthropic's approach trains a second, "replacement" model to sparsely recreate the outputs of the base model's MLP layers. This decomposes activations into sparse features. Those features turn out to correspond to high-level concepts that humans can readily identify, things like "Texas" or "the Olympics."
From there, researchers group features into causally-linked clusters by tracing how they interact during the forward pass. The result is a wiring diagram of the computation.
What the wiring diagram reveals
The reasoning chains are surprisingly legible. Ask the model "what is the capital of the state containing Dallas" and you can observe, in order: the Dallas feature activates, which triggers the Texas feature, which then triggers Austin. That is multi-step semantic inference, running on top of what looks like pseudo-symbolic reasoning.
The model also plans ahead. When generating a poem, it activates candidate rhyme features before it needs them.
This pattern appears beyond LLMs
The same phenomenon shows up in other architectures. DeepMind (2022) found that AlphaZero, trained with no human chess knowledge, independently learned intermediary representations that align with human chess concepts like "in check" and "pinning a piece." The convergence on human-interpretable concepts appears to be a property of powerful learned systems in general, not just language models.
What to do with this today
If you are building on top of LLMs and care about reliability or safety, this research trajectory is worth tracking closely. Circuit tracing techniques are moving from lab curiosity toward practical tools for steering model behavior and detecting dangerous intent before it surfaces in outputs. Right now, the actionable step is straightforward: follow Anthropic's interpretability research line, experiment with sparse feature decomposition in your own evals, and stop treating the model as a sealed unit. The internals are becoming readable, and builders who understand that early will have a real edge.