You ask ChatGPT about your own company, and what comes back is almost right. The founding year is off by one. The flagship product still carries the positioning you dropped two years ago.
And all of it is sitting on your About page, in plain text, where anyone can read it.
I do this too, and I keep landing in the same place. I once pasted a company’s own boilerplate into a chat, asked “what year was this founded?”, and got a confident answer a year off the number I’d handed over ten seconds earlier.
But the stranger part is what happens when you flip the question around. Describe your own company accurately — the category, the specialty, the customers — and ask which company that is. A person would say “oh, that sounds like so-and-so.” The model just goes quiet.
So there’s a gap between what’s in the training data and what comes back out of it. A group of researchers went after that gap with a deliberately artificial experiment, and I think what they found should change how you push your facts into the world.
Made-Up People, Memorized On Purpose
The work is a series called Physics of Language Models, from Zeyuan Allen-Zhu and colleagues at Meta FAIR (Meta’s fundamental AI research group). The name is the whole idea: physics got its laws out of stripped-down experiments, so study models the same way, under conditions you control instead of whatever the internet happens to contain.
The clever move is that they wrote a large pile of biographies for people who don’t exist. Use real celebrities and you can never rule out that the model saw the answer somewhere else; invent the people, and you know exactly what it knows — the biographies you gave it, and nothing beyond them.
Then they trained on those biographies and asked questions about them (birthdays, employers, that sort of thing), measuring how often the right answer came back as they changed the setup.
Remembering It And Getting It Back Out Aren’t The Same Thing
Here’s the first result, and it’s the one that surprised me most. When each biography was written in a single clean template, the model could barely answer questions about it at all.
Extraction, in this line of work, means pulling a memorized fact back out on demand (ie answering a question with it). And the memorizing part worked fine — the information was in there. But it just wouldn’t come when called. You know the tip-of-the-tongue feeling, where you’re sure you know the name and it won’t surface? It looks a lot like that.
So what makes it come out? What actually worked, the team found, was variety in the phrasing at training time. Same fact, different word order, different sentence shape, different surrounding context, seen again and again. Do that, and the model handles the question-form version fluently.
But when the phrasing stayed uniform, no amount of extra training afterwards rescued it. Whether a fact will be retrievable is largely settled at the moment it’s learned — that’s the claim, and it’s a blunt one.
The Model Can Barely Go Backwards
A follow-up in the series looks at what a model can do with knowledge it already holds. Some uses come easy. But others fail in a way that looks structural rather than fixable.
The easy one is straight lookup. “When is A’s birthday?” — pull the stored fact, hand it over, done. But once the task means holding several facts side by side to classify or compare them, accuracy drops off sharply unless the model gets to write out its intermediate steps first (ie think on paper before answering).
And the worst case was going backwards. Give the attributes, ask which person they belong to, and performance was close to zero. Forward works, backward collapses; it’s a door that only opens from one side. This one didn’t go away at the GPT-4 tier either.
Knowing And Answering Are Two Different Things In There
Line those up and one thread runs through them. For a model, having a piece of knowledge and being able to produce it are separate properties, not one thing under two names.
But that’s not how it works for us, which is why the mistake is so easy to make. If you know it, you can say it when asked — so we slide into treating “it’s in the training data” and “the model can answer” as the same statement. This work draws a hard line under that assumption.
The lever, and this is the part worth keeping, is how many different ways the fact showed up. A model doesn’t file one sentence away verbatim so much as it learns the fact that keeps surfacing across a lot of different wordings; that’s the version it can hand back. So the range of phrasings matters alongside the raw volume of exposure.
Conclusion: Don’t Let One Wording Do All The Work
So what does this mean if you’re the one responsible for the brand?
The first takeaway is that your important facts need to exist out there in several voices and contexts. One correct line in the company profile isn’t the finish line. News coverage, explainers, Q&A pages, other people writing about you — the same fact turning up in different clothes is, on this evidence, the condition for it becoming retrievable knowledge instead of inert stored text.
The second is about the backwards case, and it’s the one worth filing away. When someone asks “which vendors do X?” or “who in this space is known for Y?” (which, unhelpfully, is how a buyer tends to ask), that’s the direction the model is bad at, even when you genuinely fit the description. So exposure that ties your name straight to the attribute is a more realistic bet than hoping it reasons its way back to you.
Which leaves the unglamorous, load-bearing habit: check instead of assume. What a model can actually produce about you, and under which phrasings of the question, isn’t something you can work out from the outside. You measure it, change how the facts are going out, and watch whether retrieval improves. Maybe I’m wrong about how much of that is in your control — but it’s the step between the AI knowing you and the AI getting you right, and it’s cheap to look.
