Emmanuel Ameisen, an Anthropic interpretability researcher, argues that building AI models is not the same as understanding them, since they are grown through training rather than written as rules.
Ameisen says Anthropic's interpretability work finds models plan ahead, such as choosing a rhyme before writing a line of poetry, challenging the view that they simply pick the next word.
Ameisen says Anthropic's safety work rests on taking the technology's continued improvement seriously, and he argues anthropomorphic language helps predict model behavior.
Looking inside Claude reveals planning, world models and hidden quirks
Modern AI models are grown through training rather than written, so understanding them takes something closer to biology than engineering. Looking inside shows that the simple goal of predicting the next word produces models of the world, planning and shared concepts. That is why interpretability can catch problems that outputs alone hide, and why human-like descriptions of models are often useful.
Takeaways
AI models are grown, not built. Engineers design an architecture, then training on data shapes the model, so nobody wrote its internal rules. Anthropic's interpretability team studies what happens inside when a model produces a word. One model hit 12 on a math problem, then insisted on 36 in all caps.
Next-word prediction forces a model to understand the world. Predicting hard words, such as the capital of the state containing Dallas, requires real structure about the world. Ilya's example is naming the murderer at the end of a mystery novel. Models beat humans at next-word prediction around 2019.
Models plan ahead and reuse concepts internally. Researchers found a model choosing a rhyme before writing the poetry line that leads to it. Concepts such as an eye are reused across photos, drawings and text. Ameisen says next-word prediction and planning are less in conflict than they appear.
Interpretability adds a safety check beyond monitoring. Another model reading transcripts can flag bad behavior without interpretability. Looking inside found a model secretly fixated on 36 because training had rewarded a wrong answer. Golden Gate Claude showed that forcing a feature on changes behavior.
Capability jumps come from crossing reliability thresholds. Progress is smooth, yet each task becomes usable once reliability passes a threshold. After Opus 4.5, Ameisen let Claude write much of his code. Host Lenny Rachitsky reads this as a slow takeoff where problems get caught along the way.
Human-like language helps predict how models behave. Pre-training means modelling the characters who wrote the text, like an actor who must play all sides of a play. Outcome-based training then adds human-like strategies such as deception and frustration. Models are neither entities nor simple tools.
The more researchers look, the less certain they are. Ameisen says the work feels closer to biology than engineering. Models even have a gut-check mechanism that flags implausible sums, such as 23 for four plus five, without doing the math. The field has only hundreds of researchers.
How researchers test what a model is doing
Record the model's internal activations, which are lists of numbers, while it processes random text.
Tap a name to go there · tap a line to see why they are linked
On this episode 2
Show 1
Decode those activations into representations of concepts.
Form a hypothesis about what a given feature means.
Turn the feature on in every way and check whether the model behaves accordingly.