Every AI rollout I’ve watched gets stuck on the same question. “ChatGPT or Claude — which one’s better?”
Every vendor says theirs is best. The benchmark scores sit there in a neat row, one model a hair above the other. So you point at the higher number and call it a decision.
But the number doesn’t actually settle it. Put two top-ranked models into real work and the gap shows up somewhere the leaderboard never measured — one stays steady, the other wobbles on exactly the inputs you care about.
The project that takes this problem head-on is Microsoft Research’s “Physics of Language Models,” running since 2023. Its core message is blunt: today’s performance rankings are easy to misread unless you look at how the test was built.
A Benchmark Is Just A Shared Exam
Start with the word. A benchmark is a shared test — the same questions handed to every model, scored, and turned into a ranking.
That sounds clean. The trouble is what the questions are made of.
Natural-language test items drag in too many variables at once: vocabulary, context, phrasing, and the model’s prior familiarity with the material from training. When a model gets one right, you can’t easily tell whether it understood the structure of the problem or just matched a shape it had seen a thousand times before.
And there’s a nastier version of that last one. It’s called a “data leak” — when evaluation material bleeds into the training data, so the model has effectively seen the answer key before sitting the exam. The score comes back high, but it’s measuring memory, not ability.
So you end up with a ranking that looks precise and means less than it seems. The Physics of Language Models team decided the fix wasn’t a better question — it was a better way of making questions.
How Do You Build An Exam Nobody Can Cheat On?
Here’s the move. Instead of natural language, they use “synthetic data” — problems the examiner generates from scratch, where every condition is under control. No outside text to leak in, no familiarity to lean on.
Then they generate those problems with a CFG (a context-free grammar — think of it as a fixed set of grammar rules that builds each problem). Because the rules are known, every correct answer can be traced back to which piece of structural understanding produced it.
So you can finally pull the two things apart: did the model solve this because it grasped the structure, or because it pattern-matched? On a normal benchmark those two get blurred together. Here they don’t.
The name starts to make sense once you see the method. A physicist changes one variable at a time and watches what happens, so cause and effect stay legible. This is the same idea, pointed at language models — hence “Physics of Language Models.”
Why This Lands For Marketers, Not Just Researchers
I can see why this reads like a paper for AI labs. But it’s more useful at the decision-making desk than it looks.
When you’re picking an AI, the thing that matters isn’t “which model scored what.” It’s “under what conditions did that score come out.” A 92 from an exam that looks nothing like your work is worth less than an 85 from one that looks exactly like it.
That gap gets wider the more your work leans on brand context or industry jargon. Whether a model will hold up on your data depends on whether the test resembles your data — and a leaderboard number tells you nothing about that.
So if you want to know whether a model fits your job, the evaluation conditions are the thing to read, not the ranking. Skip that step and you get the classic post-rollout surprise: “this isn’t what the chart promised.”
So What: Read The Test, Not The Trophy
Pull it together and the riskiest move in AI comparison is the most common one — declaring “this model is strongest” off a score alone. What this whole line of research keeps showing is that a score is only legible next to three things: the test design, how the data was made, and how the failures were handled.
So when you’re comparing vendors, don’t stop at the scoreboard. Ask about the evaluation conditions, whether the result reproduces, and what kinds of mistakes the model actually made. A model’s failure pattern tells you more about your downside than its average score ever will.
And for checking things in-house, there’s a small move that beats any leaderboard. Fix a handful of questions that look like your real business, then repeat-measure several models under identical conditions. Same questions, same setup, run again and again.
Do that and you stop getting jerked around by a number someone else built on an exam you’ve never seen. You’re reading your own results on your own yardstick — which, in the end, is the only ranking that’s about your work.
Sources
- Zeyuan Allen-Zhu & Yuanzhi Li, “Physics of Language Models: Part 1, Learning Hierarchical Language Structures”, ICML 2023, arXiv:2305.13673
- Physics of Language Models (series site): physics.allen-zhu.com
