Do Those Extra Decimals Actually Mean Anything?
Open an AI-visibility tool or an LLM (large language model) evaluation report these days and the numbers show up suspiciously sharp. “12,847 prompts last month.” “Accuracy dropped 7.3%.” My first instinct, for years, was that the finer the digits, the more I could trust the number.
But in AI, “fine means accurate” falls apart more often than you’d think. The display runs all the way out to the decimal while the thing underneath — how the number was actually measured — never promised that much precision.
And here’s the nasty part: “looks precise, is actually coarse” doesn’t show up on the surface. Two numbers with three decimals sit next to each other, and one of them is honest while the other’s mostly costume.
So let me sort out the checks that keep you from steering on a fake decimal, using a few studies as handles.
What “Pseudo-Precision” Actually Is
The word for this is pseudo-precision. Pseudo-precision is a number that looks precise while the method behind it doesn’t actually guarantee that precision.
Picture a bathroom scale that wobbles by three kilos either way, and it flashes “62.347 kg” at you. Three decimal places, and not one of them has earned your trust. AI metrics pull the same trick.
A critical analysis from jaeckert-odaniel.com flags exactly this in “prompt search volume.” That kind of data leans on clickstream and panel estimates, and it skews toward desktop Chrome. So mobile-app usage mostly drops out of the count.
On top of that, browser-extension users skew tech-y, male, professional. Extrapolate whole-population behavior from a panel like that and the bias and error just scale up along with it.
But the screen still hands you a clean, fine-grained integer. That’s the “looks like hard data, is actually shaky” state in one picture.
Is The Model Fragile, Or Is The Scoring?
This isn’t only a search-volume problem. Search volume and LLM evaluation scores are different animals, sure. But the failure mode rhymes: trust a number without checking how it got measured, and you can get burned the same way.
In “Flaw or Artifact?”, Hua and colleagues evaluated seven major LLMs across six benchmarks and twelve templates to pin down where the “sensitivity” actually comes from.
“Prompt sensitivity” is when the same question, phrased differently, gets you a different result. What the study shows is that much of that swing is an artifact of how you measure, not a real weakness in the model. (An artifact is an apparent result your measurement method created, rather than something in the thing you’re measuring.)
Log-likelihood and exact-match scoring, it turns out, may have been blowing the gaps up bigger than they really were. Switch to LLM-as-Judge (ie letting a capable model decide whether two answers mean the same thing) and the variance dropped while ranking consistency went up.
So “the model changes its mind every time you rephrase” calms down a lot once you fix the ruler.
The Numbers That Actually Earn Your Trust
Does that mean every AI number is junk? No — come on. A well-designed metric is genuinely useful, and worth building.
Errica and colleagues propose two metrics, sensitivity and consistency. The sensitivity one measures how predictions shift under paraphrase, and — the practical bit — it works without ground-truth labels, so you don’t need a graded answer key to run it. In that work, the accuracy variation from rephrasing lands somewhere between 3.2% and 10%.
The point isn’t to memorize that range. It’s to check whether a number comes with a clear definition, reproducibility conditions, and a comparison axis. Above all, check whether the digits on display match the — the smallest step the method can actually tell apart.
When the resolution is coarser than the digits shown, the fine digits are decoration. A number whose source data, sample, and calculation you can’t see stays a directional hint, however many decimals it wears.
Conclusion: Check The Source Before You Trust The Decimal
A decimal score on an AI dashboard looks precise by default. But is it clickstream? A panel? Real logs? An experiment? Until you know, “precise-looking” is all you’ve actually got.
So the order of operations is boring, and it works: look at the source data and the reproduction conditions first. If a metric can’t explain those, it doesn’t belong as your primary KPI (key performance indicator). And in a real decision, don’t read the fancy decimal on its own — set it next to something reproducible, like your appearance rate across models or a time-series trend, and act where they line up.
None of this is an argument that AI numbers can’t be trusted. It’s the opposite: good measurement of AI visibility is real, and it’s worth doing — you just want the number whose displayed digits match what its method can genuinely see. Maybe I’m harder on decimals than they deserve. But a number that won’t show its work isn’t evidence; it’s a confident-looking digit.
If you want the concrete version of this trap, ‘Prompt Search Volume’ Looks Precise. That’s The Problem. walks through one metric end to end.
Sources
- jaeckert-odaniel.com, “Prompt search volume: Real data or all guessed?”, 2025-12-16, jaeckert-odaniel.com
- Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, Yao Qin, “Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs”, EMNLP 2025 Main Conference, arXiv:2509.01790
- Federico Errica, Giuseppe Siracusano, Davide Sanvito, Roberto Bifulco, “What Did I Do Wrong? Quantifying LLMs’ Sensitivity and Consistency to Prompt Engineering”, NAACL 2025, arXiv:2406.12334
Related
- ‘Prompt Search Volume’ Looks Precise. That’s The Problem. — where the panel behind the number comes from, and how far you can lean on it
- Is The AI Actually Inconsistent, Or Just Badly Measured? — when the “swing” is the measuring stick, not the model
