Somebody on the call says “we’re on the latest model”, and you can feel the room relax.
I do it too. Put two tools side by side in a comparison table and my eye goes straight to whichever one has the bigger number next to it.
But size turns out to be a stranger variable than the spec sheet makes it look. It matters — just not on its own, and not always in the direction you’d guess.
The research that sorts this out goes by the name “scaling laws”: work from OpenAI, DeepMind, and a Google Research team. Between them they hand you a decent way to read anyone’s AI performance claims.
So let’s look at what they found.
Make It Bigger And It Really Does Get Better
The starting point is a 2020 paper from Kaplan and colleagues at OpenAI, “Scaling Laws for Neural Language Models”.
They showed that a language model’s loss follows a power law against three inputs: model size, dataset size, and the compute spent on training. Loss is basically the model’s error at predicting the next piece of text, and a power law just means the relationship is regular — push an input up, watch the error come down in a plottable way rather than a lucky one.
And it held across more than seven orders of magnitude. The reported scaling exponent is about 0.076 for model size and about 0.095 for dataset size, and bigger models turned out to reach the same loss on less data.
The exponents don’t matter for your job. What they imply does: gains from scale are, roughly, forecastable.
That’s the finding that turned “just make it bigger” from a hope into a plan.
But A Big Model On A Small Diet Is Undertrained
Then the correction arrived, fast. DeepMind’s Hoffmann and colleagues published the Chinchilla paper in 2022, “Training Compute-Optimal Large Language Models”.
They trained more than 400 language models, from 70 million to 16 billion parameters, on 5 billion to 500 billion tokens. (A token is the small chunk a model chews text in — roughly a word or a piece of one.)
So here’s what came out: for a fixed compute budget, model size and token count should scale roughly in step. Double the model, double the training data.
Chinchilla was built on that principle at 70 billion parameters. And it beat the existing models — including Gopher, four times its size at 280 billion — across a wide range of evaluation tasks.
The picture I keep coming back to is the new hire with the enormous résumé and no reps. The title on the card is real. The field experience isn’t there yet, and it shows up in exactly the situations you hired them for.
Two Different Reasons A Model Stops Improving
There’s one more piece, and it’s the one that earns its keep in meetings. Bahri and colleagues at Google Research and Johns Hopkins University went after the why: what actually produces these scaling laws? Their answer landed in a 2024 PNAS paper, “Explaining Neural Scaling Laws”.
They split the origins of scaling into two kinds:
- Variance-limited — the bottleneck is statistical noise from having only a finite sample of data.
- Resolution-limited — the bottleneck is the model’s own capacity to represent the world finely enough.
Put plainly: is the answer wobbling because there isn’t enough data, or because the model can’t see in enough detail? Same symptom, different cure.
And this is exactly where I catch myself making the wrong move. When an AI tool gives a weak answer about our category, my first instinct is “get a stronger model”. But maybe the evaluation set was lopsided, or the tool never got the context that would make the question answerable. Swapping models fixes neither of those.
What To Look At Instead Of The Model Name
So what does a marketer or a brand lead actually do with this?
Mostly: stop letting the model name carry the whole performance argument.
To be fair, scaling laws are general findings about training, not a verdict on any particular product; past this point I’m reading them as a hint about how to evaluate tools, not as direct evidence about them. Still, when you’re using AI to see how your brand gets talked about, “generally smart” isn’t what you’re buying. You want answers that hold up in your category, against your comparison set, for your customers. Which means general capability and fit-for-purpose need separate scores.
Three things worth pulling apart the next time someone walks you through a demo:
- which model is underneath it
- what data and context the evaluation ran on
- which metrics counted as good or bad for that particular use
Conclusion: Ask What Was Measured, Not Which Model Ran It
Scaling laws do say that bigger tends to be better. But Chinchilla showed the efficiency falls over when size and data drift out of balance, and the PNAS work says a plateau can come from thin data or from limited capacity — two different problems with two different fixes. “It’s the latest model, so we’re fine” is a pretty coarse way to decide anything.
Maybe I’m underrating how much raw capability still buys you; the honest answer is that it varies by task. But the question that changes your decision isn’t which model a tool runs. It’s what data it was tested on, what got measured, and how far the testing went. For brand work, that means asking whether it gives sensible answers about your category and your competitors — not whether it can write a passable sonnet.
So next time a comparison table lands in front of you, don’t stop at the “which AI” column. Read across to what was measured, and on what.
Sources
