Start using AI at work and the same worry shows up, every time, in every team. “If I phrase it slightly wrong, the answer turns into garbage.” “We’d better memorize the correct prompt templates or we’re in trouble.”
I get it, because I’ve felt it. There’s a real urge to hand around a prompt cheat-sheet and standardize the whole company on it.
But here’s where I want to stop for a second. Before we pile on the operating cost — the templates, the training, the prompt police — has anyone actually checked how much the answer wobbles? Because “it must wobble a lot” is an assumption, not a measurement.
So let’s look at two studies. One put a number on how sensitive models really are. The other asked whether that sensitivity is even real, or something our measuring stick invented.
”If I Ask It Slightly Wrong, Is The Whole Thing Ruined?”
First, the obvious question: when you reword a prompt, how much does the answer actually change? Not “does it change at all” — of course it does a little — but by how much.
A team led by Errica got concrete about this (Errica et al., NAACL 2025). They set up two measures to pin it down:
- Sensitivity: how much the prediction shifts when you reword the prompt
- Consistency: how stable the predictions stay across rewordings of the same inputs
The rewording here is basically paraphrasing — same meaning, different words. Think of it as saying the exact same thing two different ways and seeing if the model flinches.
And across a batch of classification tasks, the accuracy swing came in at 3.2 to 10%. Not zero. But nowhere near “one wrong word and it all collapses.” A lot of people treat their prompts like a bomb squad treats a red wire — and it turns out the wire mostly doesn’t blow.
There’s a more useful finding buried in there, though. What moved the needle wasn’t surface word order — it was the underlying meaning. So the thing worth getting right isn’t the magic phrasing. It’s being clear about what you actually want the model to decide.
Maybe The AI Isn’t Flaky — Maybe Our Ruler Is Bent
Now the sharper question, from a study by Hua and colleagues (Hua et al., EMNLP 2025). They asked: is high sensitivity a weakness of the model, or an illusion the evaluation method cooks up? They checked seven major LLMs across six benchmarks and twelve templates.
Two of the old standby scoring methods went on trial here:
- Log-likelihood scoring (ie grading by the probability the model assigns to each answer, not the answer it actually gives)
- Exact-string matching (ie the answer counts only if it matches the key character-for-character)
Both of them, it turns out, flag answers that mean the same thing as mismatches — and that makes the model look way more jittery than it is. Swap in LLM-as-Judge (ie having a capable model decide whether two answers mean the same thing) and the performance variance dropped, while the rankings got more consistent.
Which means a chunk of “the AI is unstable” was never the AI. It was the ruler. And if your ruler is bent, every measurement off it is wrong in the same flattering, alarming way.
Conclusion: Check Your Grading Before You Chase The Magic Phrase
Prompts do change results — that part’s true. But spending your life hunting for the one incantation that wins isn’t where the leverage is. What these two studies show is that a good slice of “the prompt is the problem” was actually the coarseness of the scoring and the exact-match reflex puffing things up.
So the first thing to audit in your own setup isn’t the wording — it’s how you’re grading the answers. Are you marking things wrong on an exact-match basis when the meaning was fine? Are you throwing out perfectly good answers because they didn’t hit the expected string? Fix that, and a lot of the apparent wobble just settles down.
After that, your internal prompt policy can be refreshingly small: say the goal, the constraints, and the output format you want, and stop. Maybe I’m wrong about the exact split — but in that order, the “if I ask it wrong, it’s all ruined” dread tends to quietly let go of you.
Source
- Errica, F., Siracusano, G., Sanvito, D., & Bifulco, R., “What Did I Do Wrong? Quantifying LLMs’ Sensitivity and Consistency to Prompt Engineering”, NAACL 2025, arXiv:2406.12334
- Hua, A., Tang, K., Gu, C., Gu, J., Wong, E., & Qin, Y., “Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs”, EMNLP 2025, arXiv:2509.01790
