“Is the AI’s answer the result of actually understanding the thing?”
If you use AI at work, you can’t really dodge this one. The output reads smooth — but did it grasp the content, or is it just stringing together something plausible?
The better the result, the more the doubt creeps in. Hand it a first-draft plan or an analysis memo and the worry sharpens: maybe this is all surface and no grip.
And the question isn’t idle. How much you trust an AI with depends on the answer, which means it lands right on the tool you pick.
In the previous piece, I wrote about how the “Physics of Language Models” project rebuilds the test so a score is actually legible. The same Microsoft Research team — Zeyuan Allen-Zhu and Yuanzhi Li — went one step further and looked inside the model while it reads. So let’s see what they found.
What’s The Model Actually Holding In Its Head?
First, the setup, because it’s what makes the rest trustworthy. The team tested models on structured synthetic data — problems built from a fixed set of rules, so it’s always clear what’s being measured. No outside text sneaking in, no familiarity to lean on.
Then they watched the “hidden states.” A hidden state is basically the model’s internal scratchpad — the working representation it holds while reading a sentence, before it commits to an answer. Think of it as the notes you’d jot in the margin while parsing a tricky clause.
So what was on the scratchpad? For sentences with nested structure (a clause inside a clause inside a clause), the analysis found the model had reconstructed a hierarchy of relationships inside itself. It wasn’t tracking a flat row of words. It was tracking the sentence’s blueprint — which part belongs under which.
That’s a bigger deal than it sounds. “Reads the words” and “rebuilds the structure” are different abilities, and only the second one travels to messy real inputs.
The Attention Looked Like It Was Following A Plan
The next question is how it built that blueprint. So the team traced the model’s attention — which earlier words it leans on as it processes each new one.
And the pattern looked a lot like — a problem-solving move where you split a big problem into smaller pieces, solve each once, and reuse those partial answers instead of redoing the work. Which, for you, means the model wasn’t just guessing the next word; it had something like an internal order of operations.
That’s the heart of the “does it understand” question. Guessing and planning can produce the same fluent sentence on an easy input. They come apart on a hard one — and what the attention trace shows is closer to planning.
I’d hold this one a little loosely. It’s a careful study on controlled data, not a guarantee about every model on your real prompts. But “the model builds an internal procedure” is a much better starting bet than “it pattern-matches and hopes.”
And The Design Of The Model Changes How Deep That Goes
Here’s the catch, though: not every model does this equally well. How the model is built moves the needle on how much real structure it picks up.
Take how a model tracks word position. Some designs hard-code each slot (“this is word number 7”); others, like RoPE, encode relative position (“this word is three back from that one”). In the tests, the relative-position approach came out ahead at holding nested structure together.
The architecture mattered too. Comparing a GPT-type model (it reads strictly left to right) against a BERT-type one (it sees the whole sentence at once), the more constrained design showed deeper structural understanding in some cases — the tighter rules seem to push the model toward a real procedure instead of a shortcut. So “same score, same model” doesn’t hold. Two models can land on the same leaderboard number and process the input completely differently underneath.
Conclusion: Judge The Failures, Not The Fluency
If one thing here changes how you buy AI, it’s this: an answer that looks fluent and a model that’s processing stably are two different things. Clean prose makes you want to relax. But you don’t know the real strength until you see where it cracks — on long passages, on comparisons, on anything with a branch or a condition in it. That’s exactly where the planning-versus-guessing gap shows up.
So when you’re comparing candidate models, write down the kinds of questions that failed, not just the accuracy number. A failure pattern tells you more about your downside than an average score ever will. When a vendor walks you through their model, push past the showcase wins — ask which conditions it’s weak on and what the test conditions actually were. That’s the part that’s load-bearing for your decision.
And before you commit, re-run it on your own task. Not the demo, not the benchmark — a handful of questions that look like your real work, under your real conditions. Maybe it holds up and maybe it doesn’t, but at least you’ll be reading your own results instead of trusting a number someone else built on an exam you’ve never seen.
Sources
- Zeyuan Allen-Zhu & Yuanzhi Li, “Physics of Language Models: Part 1, Learning Hierarchical Language Structures”, ICML 2023, arXiv:2305.13673
- Physics of Language Models (series site): physics.allen-zhu.com
