“I Pasted In Two Old Emails And It Got Weirdly Good”
You ask the AI for something, it comes back mediocre, so you paste in two or three examples of what you actually want — and the second answer is noticeably better.
That’s probably happened to you at some point in the last few months.
It’s happened to me a lot. When I ask for an email draft, sticking a few of our old ones into the prompt first changes the output in a way I can see immediately.
But then the doubt arrives, right on cue: is it really learning from those examples, or did I just get lucky that time? It feels like it works. And “feels like it works” is a shaky thing to build a team habit on, because you never know whether the trick will hold tomorrow.
So it’s worth knowing that this has a name and a proof behind it. Let’s look at the paper that went after it head-on.
The Trick Has A Name, And So Does What The Model Is Doing
First, one bit of vocabulary. Pasting examples into the prompt is called few-shot prompting — “few-shot” just meaning you show it a handful of examples. No examples at all is zero-shot; two or three is few-shot. That’s the whole distinction.
And the thing happening inside the model when you do it is . In-context learning is basically this: the model adapts its behaviour to the examples sitting in your prompt, with no extra training involved. Nothing inside the model gets rewritten — its parameters are exactly what they were a second ago — and yet it answers differently because it saw your examples. Which, for you, means the leverage is in the prompt itself rather than in waiting for a better model.
But the open question was whether that deserves to be called learning at all, or whether it’s just a convincing impression of it.
What UC Berkeley Proved
A team at the University of California, Berkeley — Ruiqi Zhang, Spencer Frei and Peter L. Bartlett — went at exactly that question, and published the result in 2024 in JMLR, one of the main machine learning journals (paper).
What they did was build a stripped-down model of a Transformer (the architecture underneath today’s AI) and follow the training process through with actual mathematics. Real models are too tangled to solve exactly, so you keep only the essentials and work out what’s going on inside the simplest version that still behaves like the real thing.
So here’s what came out:
- Train it long enough and the model reliably reaches its best possible state (convergence to the global minimum)
- In that state, its in-context learning is as accurate as the theoretically best predictor for the data in front of it
Put plainly: the moment the model reads your examples, it assembles a prediction rule that fits those examples as well as anything could. It’s rock-paper-scissors where you get to throw after seeing the other hand — which is, honestly, pretty sharp.
Why This Counts As Studying, Not Faking It
The genuinely fun part comes next. The team also worked out that the computation the model runs internally amounts to the same thing as the familiar machine-learning routine of nudging a model closer to right, one example at a time.
Show it examples, then, and it runs something like a fresh round of learning-the-rule-from-data, right there in the prompt (no training run, no fine-tuning, nothing rewritten inside the model). So no, the sense that it’s studying isn’t a projection you’re putting onto it. Examples work because of how the thing is built.
The Catch: This Is A Proof About A Linear World
Two caveats, and both matter.
The proof holds in a deliberately simplified, linear setting. But nobody can yet claim that the identical thing happens, unchanged, inside something as large and layered as ChatGPT. The authors say as much themselves — extending it to messier settings is future work. One very good lamp is now lit at the mouth of a very big cave; the far end is still dark.
The second caveat is the one that bites in daily use: in-context learning is strictly a one-session effect. Whatever the model picked up from your examples is gone when the prompt closes. It doesn’t retain any of it (which, if you’ve ever wondered why yesterday’s brilliant chat is no help this morning, is your answer).
Conclusion: Keep The Examples, Because The Model Won’t
Pull it back to the day job and it lands somewhere simple. Pasting examples into a prompt isn’t a superstition or a good-luck charm; it’s a sound move, grounded in how these systems work. The Berkeley result says the model assembles the best answering rule it can from whatever examples it can see. So handing it one or two that look like the output you want is, on the math, exactly the right thing to do.
What that means operationally is one habit: stop treating your good examples as somebody’s personal touch. If your team shares AI work at all, the winning examples belong in a template that gets pasted in every single time — because today’s excellent prompt doesn’t carry over to tomorrow’s new chat at all.
And it’s worth holding the line between two different things. In-context learning handles on-the-spot adjustment. Making a model actually hold your company’s information and rules permanently is a separate problem with separate machinery, and prompt craft won’t get you there. Keep those apart and the next “should we just prompt our way out of this?” conversation gets a lot shorter.
None of this is settled — the field is young. But “showing it examples works” turns out to be a well-founded claim, so I’d start by adding one example to a prompt you already use and seeing what moves.
Source
- Ruiqi Zhang, Spencer Frei, Peter L. Bartlett (UC Berkeley), “Trained Transformers Learn Linear Models In-Context”, JMLR Vol.25, 2024, link
