More and more, before building a pitch deck, people paste a handful of competitors into ChatGPT and ask “which of these is strongest?” or “just rank these for me.” I’ve watched colleagues do it minutes before a meeting — dump the candidate list in, let the model sort it out.
And then the ranking that comes back gets pasted straight into the deck as “the AI’s objective take.”
But here’s the thing that nags at me. If you paste the same names in a different order, do you get the same ranking?
You’ve probably felt this once — “wait, that’s not the order it gave me last time.” It turns out someone measured exactly how big that wobble is, and the number is larger than I’d have guessed. So let’s look at it.
What “The Order Changes The Answer” Actually Means
First, the plain version. Position bias is just this: the order you show the candidates in changes which one the model picks. You want it judging on the merits, but the slot each option sits in tugs the score around.
People do this too. Give someone three options in a row and the middle one somehow looks like the safe pick. (It’s the same reflex that has you grabbing the product on the middle shelf at the supermarket.) The model has its own version of that pull, and it quietly eats into how stable its recommendations are.
The real question is how big the wobble is. If it’s tiny, fine, ignore it. But if it isn’t, then asking an AI to rank things is a shakier move than it looks.
An RMIT Team Reshuffled The List And Measured The Wobble
That’s the question Ethan Bito and colleagues took head-on (the paper). They had LLaMA 3.3 70B — a big model — recommend movies and books.
They used two standard datasets: MovieLens for film ratings, and Amazon’s book ratings. The method was refreshingly plain. Take the same set of candidates, shuffle the order, ask for a recommendation again, and measure how much the result holds together across shuffles.
The yardstick was a “consistency score”: 1.0 means the ranking never budges when you reorder, 0 means it comes out different every time.
So Here’s What They Found
The longer the candidate list, the more the consistency fell apart.
- Movies: 0.67 consistency with 10 candidates, dropping to 0.47 at 30 (a 0.20-point fall, about 30%).
- Books: 0.55 at 10 candidates, down to 0.47 at 30 (about 15%).
So the more options you make the AI weigh, the more “the paste-order decides the ranking.” At ten candidates the movie result holds together about two-thirds of the time; push it to thirty and nearly half of it is just getting dragged around by position. Pile on candidates, and the judgment goes mushy.
The authors tie this to “lost in the middle” — the known habit of models to drop information sitting in the middle of a long input. A candidate parked halfway down the list gets treated as fainter than it is. Which, if you’re the brand sitting in slot 15, isn’t a comforting thought.
Change How You Ask, And The Wobble Shrinks
Now the part that keeps this from being all doom. The team also tried a different way of asking, called RISE. Instead of dumping the whole list in and asking for one ranking, you have the model pick one, set it aside, pick the next, and repeat.
Switch to that, and consistency at 30 candidates climbs back up: movies from 0.47 to 0.69 (about 46% better than the standard ask), books from 0.47 to 0.65 (about 38% better).
The point is they didn’t rebuild the model. They just changed the question. Which is good news — but it also tells you how fragile the plain “rank all of these” request was to start with.
Conclusion: Treat An AI Ranking As One Draw, Not The Verdict
So the takeaway isn’t “AI can’t judge.” It’s that a single “recommended order” from a model is one roll of the dice, not a measurement — and the paste-order alone can move it by about a third.
There are two small things I’d change in how you use it. The first: shuffle and ask more than once. Instead of celebrating the one run where you came first, reorder the list, run it a few times, and see whether you’re steadily near the top. That alone keeps you from getting fooled by a lucky arrangement.
The second: judge fewer things at once. Ranking 30 items in one go is exactly where position bias is worst, so narrow it to the five or ten that actually matter. When you’ve got a lot of angles to compare, split them across separate asks — the answers hold up better.
You’ll be handing more of these judgments to AI over time, and just knowing that “the order can move the answer” changes how much weight you put on any one ranking. Next time it matters, shuffle your own list and ask three times before you believe it.
Source
- Ethan Bito, Yongli Ren, Estrid He (2025), “Evaluating Position Bias in Large Language Model Recommendations”, arXiv:2508.02020, arXiv
