“How Much Of Our Field Can We Actually Hand To The AI?”
The further you get into using AI, the heavier that question gets. You poured your own product specs and industry know-how into the model, and the answer that comes back still misses.
I had the same naive start — “feed it the data and it’ll answer.” In practice, “it should know this but it won’t say it” happens constantly.
The work that pulls that gap apart with controlled experiments is Physics of Language Models, Part 3 (3.1 through 3.3).
The takeaway: there are three separate walls. How you teach it, how you ask it, and how much it can hold.
Wall 1: One Fact In One Phrasing Doesn’t Stick
The first finding is pleasingly practical. The AI’s recall drops when the same fact only ever appears in a single, fixed wording.
The researchers used synthetic biography data and compared whether the same fact was given once, or repeated in different phrasings and orders.
The result was clean. A single phrasing alone barely moved accuracy. Mixing in paraphrases and reordered versions pushed extraction accuracy up.
That’s exactly why “drop the source text into RAG once and call it done” is weak. A summary version, an FAQ version, a bulleted version — presenting the same thing from several angles is what does the work.
Wall 2: Flip The Direction Of The Question And Accuracy Collapses
The second wall is the direction you ask from. The model handles a forward question fine, but a tends to make accuracy fall off.
Think “Which university did A attend?” — answered easily — versus “Who went to Tokyo University?” — where it falls apart. Same fact, opposite direction.
This bites quietly in support and product recommendation work. When the real questions lead with the condition rather than the subject, you’ll leak answers unless you design for it.
Wall 3: There’s A Ceiling On How Much It Can Know
The third wall is capacity. Part 3.3 shows a scaling law linking parameter count to the amount of knowledge a model can hold.
In short: bigger does not mean infinitely more memory. Scaling the model up raises capacity, but it scales straight into cost.
So instead of betting everything on cramming it all into the model itself, splitting work into a RAG setup and a defined scope of coverage starts to matter a lot.
Conclusion: Start Your RAG Work With “More Phrasings” And “Question Rewriting”
When the AI won’t answer from knowledge you handed it, jumping straight to “the model is weak” is premature. As this research shows, the issue isn’t only how much it knows — it’s whether the stored phrasing and the phrasing your users ask with actually line up.
For RAG and internal knowledge bases, don’t register a key fact as one single sentence. Hold it in several shapes — a summary, an FAQ, a bulleted list. And when a live question fails, try rewriting the question itself into a plain forward confirmation and asking again. These unglamorous tweaks often pay off before swapping the model ever would.
Sources
- Zeyuan Allen-Zhu, Yuanzhi Li, “Physics of Language Models: Part 3.1, Knowledge Storage and Extraction”, arXiv:2309.14316, 2023. paper
- Zeyuan Allen-Zhu, Yuanzhi Li, “Physics of Language Models: Part 3.2, Knowledge Manipulation”, arXiv:2309.14402, 2023. paper
- Zeyuan Allen-Zhu, Yuanzhi Li, “Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws”, ICLR 2025. paper
- Physics of Language Models (series site): physics.allen-zhu.com
Related
- When You Tell The AI To “Think,” Is It Really Thinking? — the real reach and limits of reasoning
