“The new model is just smarter.” You’ve heard that line about four times this year, right?
I nod along too. And then I catch myself, because I couldn’t tell you what “smarter” is pointing at.
We get to see the outcome. Better answers, better scores. The middle bit — the stretch where the thing is actually becoming smarter — stays shut.
But a few people have started prying that middle open. Not dissecting a finished model; following one while it grows.
So let’s look at what they found, which is that the parts inside a model quietly divide up the work while it trains.
”Sure, But What’s Going On In There While It Learns?”
The piece at the center of this is a paper from Timaeus, a research outfit, accepted as a Spotlight at ICLR 2025: “Differentiation and Specialization in LLMs” (the research page).
What they went after is a part called the attention head. An attention head is the bit that decides which words the model looks at while it reads a sentence, and a single model stacks a lot of them.
The interesting thing is how those heads look on day one.
Right after training starts, they all behave more or less the same way. There’s barely any difference in what they do.
The paper calls that state “undifferentiated.” And as training piles up, the heads drift into different jobs. Pretty dutiful growing-up, honestly.
It’s Embryology, Except The Embryo Is A Model
“Starts out identical, then splits into roles” — you’ve met this story before, haven’t you.
It’s how living things develop. One fertilized egg divides and divides, and eventually some cells become heart and some become nerve.
Think of the model’s parts as a graduating class that all looked interchangeable in week one, and after training some of them end up in sales and some in accounting. Not far off.
So what do the differentiated heads actually specialize in? A couple of the observed roles:
- A simple one that predicts the next word from the word right before it (bigram, ie handling the link between two adjacent words)
- One that spots a pattern from earlier in the text and applies it again (induction, ie repeating a rule found in context)
On top of that, they turned up something nobody had described before: “multigram circuits,” which handle slightly more complicated patterns.
That’s the payoff for following the training run. Structure you’d never notice by staring at the finished thing.
How Do You Measure A Part Getting Good At Its Job?
Fair question at this point: how on earth do you measure who’s doing what inside there? This is the clever part.
They used a metric built on singular learning theory — a mathematical framework, developed by Sumio Watanabe, for rigorously analyzing learning in models that ordinary statistics handles badly, neural networks very much included.
Apply that metric head by head, and you get a number for “how far along is this particular part right now.”
And here’s the part that matters. None of this shows up if you’re only watching the model’s grades.
Normally you gauge a training model’s smarts by how the test error (the loss) falls. But even while that curve is gliding down nice and smooth, a staged structural change — the division of labor — is going on underneath, unannounced.
So instead of a static autopsy of a finished model, you follow the changes along the axis of training time. The paper calls this .
Interpretability research with one extra axis bolted on: time. That’s the shortest way I can put it.
Conclusion: Ask Whether The AI Can Be Explained, Not Just Scored
What the paper shows is that a model’s smarts don’t just accumulate smoothly. They arrive as a staged process, with parts inside getting assigned their roles one after another.
Bits that started out flat and interchangeable turn into specialists through training, and you can now watch that happen somewhere other than the score column. Which I think is a bigger deal than it sounds.
Now, the honest caveat: what got confirmed here was on small models, roughly two layers. Nobody gets to claim the same thing happens inside the giant ones. Discount accordingly.
But the thing worth carrying back, if you’re a marketer or the person signing off on tools, is that “AI is a black box, so all you can do is compare final performance scores” is starting to crack.
As research that can ask what’s going on inside keeps growing up, so does our odds of checking why a given model decided what it decided.
So when you’re sizing up an AI or picking between two, don’t swallow the scores you’re handed. “Can anybody explain this thing’s insides?” is a question with a rising exchange rate, and it costs nothing to keep it in your back pocket.
Honestly, I didn’t expect this much to fall out of watching a model grow up. If it caught you too, start small: next time an answer surprises you, stop and ask why that one came out.
Sources
- Timaeus Research, “Differentiation and Specialization in LLMs”, ICLR 2025 (Spotlight), 2024, link
