“Come On, It’s Just Stringing Plausible Words Together”
Somebody says this every time AI comes up in a meeting. The model doesn’t understand anything, it’s just picking likely words by probability, and everyone nods.
I was in that camp too (and if you’ve ever quietly agreed in a meeting while planning to use the thing anyway, hello). Math especially — anything with steps you have to get in the right order — felt like the place a language model would get caught out, pattern-matching against answers it had seen somewhere.
But if it were pure memorization, unfamiliar problems should break it completely. And they don’t. Hand it a word problem it’s never seen and it does fine, mostly.
So what is actually happening in there?
A few researchers went at this the only way you really can: by building the test problems themselves. Let’s look at what turned up.
Problems The Model Couldn’t Possibly Have Read
The work comes out of the “Physics of Language Models” series that Zeyuan Allen-Zhu and colleagues run at Meta FAIR (R). The idea is borrowed from physics: rather than argue about what a model “really” does, build an experiment controlled enough to give a clean answer.
The clever bit is the data. They generated a big pile of grade-school math word problems from scratch instead of scraping any off the internet.
That matters more than it sounds. With problems pulled off the web, you can never rule out that the model read the answer during training; with problems that didn’t exist until the moment somebody made them, you can. So if it solves one, it solved it. That’s not memory, it’s work done on the spot.
Template Memorization Doesn’t Explain It
And what came back is that the model’s method isn’t template memorization.
By “reasoning process” I just mean the intermediate steps it goes through before committing to an answer (the scratch work, basically). When the team looked inside the model, they found something like planning: once it’s read the problem, it’s already estimating which quantities the answer is going to need, before it writes anything down.
That’s not slotting numbers into a memorized formula. That’s building a route, problem by problem.
But the route isn’t the one you’d take. The model lines up quantities in an order no person would use, and it works out some numbers the problem doesn’t need at all — running ahead, computing things just in case. Think of a cook who chops every vegetable in the kitchen before deciding what dinner is. Same plate at the end, and a genuinely alien way of getting there.
Show It A Mistake And It Gets Better
The follow-up paper asks the practical question: how do you train this math model so it’s more accurate (R)? The answer is a bit backwards.
You’d assume the worked examples should be clean. Show it right answers, get right answers.
But when the training data deliberately mixed in flows where the model got something wrong and then immediately corrected itself, reasoning accuracy came out higher than it did on correct solutions alone.
And it took no elaborate machinery — no multi-round setup where you make it answer, tell it off, and make it try again. Slipping correction examples into ordinary next-sentence-prediction training was enough on its own. Failures weren’t waste. They were teaching material, which I find genuinely delightful.
Thinking, But Not Our Kind Of Thinking
Put the two papers side by side and a line shows up. The model isn’t “just memorizing”. It also isn’t thinking the way you do.
There’s real planning in there, which is why an unfamiliar problem doesn’t break it. But that plan has its own habits, and they don’t line up with human intuition. “Smart” and “human-like” are worth keeping in separate boxes.
And that gap — thinking, but differently — is the part that reaches your desk.
Conclusion: Check The Answer, Not How Good The Reasoning Looks
Two things to take away, and I’d stop at two.
First, don’t accept an AI’s output because it showed its work. The planning is real. But it’s planning with unfamiliar habits, and a plausible-looking process guarantees nothing about the claim at the end of it. So the thing to check isn’t how smooth the explanation reads, it’s whether the final answer matches reality — with numbers and proper nouns especially, a natural delivery buys you nothing. Look it up once.
Second, when you’re getting an AI to fix something, hand it the mistake and the fix together. “Write it like this” gives it the destination; “this bit is wrong, here’s what it should say instead” gives it the route, and the route is what it rebuilds from. Criticism lands better with the repair attached.
Maybe I’m reading too much into two papers about grade-school arithmetic. But the direction feels right: AI does look smart, and I think it genuinely is — just not as a longer version of us. Which is why the boring habit, leaning on it happily and still checking the final claim yourself, is the thing that makes the relationship last.
