Somewhere in an agency deck, or in your own team’s Slack, there’s a line that goes: for AI, step one is putting up an llms.txt file.
It usually comes with a one-line gloss — it’s robots.txt, but for AI — and at that point there’s no obvious reason to argue.
It’s one file. It costs basically nothing. I put one up myself, on the theory that it couldn’t hurt (the same theory that governs the emergency kit in my closet, which I have never once opened).
But there’s a plain question sitting underneath all this.
Once the file is up there, does anything actually come and read it?
Somebody finally answered that with server logs from 137,000 sites. So let’s look.
So What Is llms.txt Supposed To Do, Again?
is a text file you park at the root of your site — the top level, where robots.txt lives — that lists your important pages and summaries. Which, for you, means it’s a note to the models saying read this part, not the rest.
It was proposed in 2024, and the pitch is a site map written for AI rather than for Google.
Here’s the catch, though. No major AI platform has committed to reading it, and Google has gone further, saying outright that its search systems don’t use llms.txt.
So nothing in the plumbing turns “I put it up” into “it got read.” That’s the gap the logs fill.
137,000 Sites’ Worth Of Server Logs
Ahrefs, the SEO toolmaker, ran the analysis and put it out in June 2026 (the study).
They took every domain in their measurement network with traffic in May 2026 — 137,210 of them — and went to the . Server logs are just the raw record of who came and asked for which file: not a model, not an estimate, an actual list of requests. Which makes them close to ideal for the question “did anyone read this?”
Worth naming the conflict first: Ahrefs sells SEO tools, so “classic SEO still works” is a convenient conclusion for them. They also flag that their measurement skews toward sites running their products, which makes the 28% adoption figure an upper bound by their own admission. With that on the table, the numbers:
- Domains publishing a valid llms.txt: 28% (about 38,360)
- Of those, share with zero requests in May 2026: 97%
- Share that got any request at all: 3% (about 1,100)
So for more than nine files in ten, nothing came by that month. Not once.
Who Was Actually Fetching That 3%?
Now the fun part — the small slice that did get traffic. Of those requests, 96% came from bots; humans made up 4%.
And of the bots, only 19.5% could be pinned down as AI-related. The other 77% were tools with nothing to do with AI, and 12% of those were SEO crawlers checking whether the llms.txt file was installed correctly. Which is a neat little loop: the file’s main audience turns out to be the software that checks the file exists.
Open up the AI slice and it breaks down like this:
- AI agents and agent infrastructure: 10.5%
- AI training crawlers: 5.3%
- AI assistants: 2.5%
- AI search fetch bots: 1.1%
Named bots: GPTBot at 4.51%, ClaudeBot at 0.80%. But the number that settled it for me is this one.
On sites without an llms.txt, the number of AI bots that came looking for one was zero.
Nobody’s knocking. And if nobody’s knocking, putting up a door doesn’t get you a visit.
Meanwhile, The Files Themselves Went Up 8.8x
The supply side is moving in exactly the opposite direction.
Originality.ai tracked more than three million sites over twelve months and watched llms.txt installs climb from 4,088 to 36,120 — 8.8x in a year.
The cousins (llms-full.txt, ai.txt) grew around a hundredfold, though from bases so small the multiple flatters them.
So expectations on the publishing side have run well ahead of anything the engines committed to. That’s a familiar shape; it’s just unusually easy to see in these numbers.
If I stopped here this would be a bummer of a post. But the same logs have a second half, and I think that’s the part worth your time.
Where AI Actually Comes To Read
Everything above is about a file nobody fetched. Here’s the one everybody fetched: the same analysis found AI search crawlers pulling site directly.
HTML is just the page itself — the thing a person sees in a browser. So the models are going to the same place your customers do, which moves the whole question. If you want AI to read something, the place to put it isn’t a side file; it’s the page.
That makes the checklist pretty concrete:
- Can your main pages be fetched at all?
- Is what’s on them structured so a model can read it?
- Is anyone outside your own domain talking about you?
We’ve covered the first two with real numbers before: roughly a third of product pages turned out to be invisible to AI shopping traffic, and adding sources and stats moved citation odds by 30-40%. Against the logs, tidying up what you already publish looks like the shorter path.
Conclusion: Fix The Pages You’ve Got, Then Watch What Moves
None of this makes llms.txt a mistake. It’s a reasonable idea the engines haven’t picked up — yet, maybe. But the crawlers are at your HTML right now (the logs say so), and attention is finite, so that’s where I’d spend it.
Start by checking that your most important pages can actually be fetched and read. Before hunting for a new file to put out for the AI, make sure the pages already out there come through.
Then watch whether it moves anything: how often your name shows up in AI answers, which of your pages get cited. Maybe I’m wrong about how long llms.txt stays ignored — but you’ll only find out either way if you’re measuring the output.
Boring work, mostly. Boring work is usually the stuff that pays.
Sources
- Ahrefs (Louise Linehan), “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read”, June 15 2026, ahrefs.com (Server logs from Ahrefs Web Analytics / Bot Analytics covering 137,210 domains with traffic in May 2026. After verifying that a root llms.txt returned HTTP 200 with actual Markdown, 28% (about 38,360 domains) published a valid file; 97% of those received zero requests in May 2026 and 3% (about 1,100) received any. Of that 3%: 96% bots, 4% humans; of the bots, 19.5% AI-related and 77% non-AI tools, 12% of which were SEO tools auditing llms.txt. AI breakdown: AI agents/agent infrastructure 10.5%, AI training crawlers 5.3%, AI assistants 2.5%, AI search fetch bots 1.1%. Named bots: GPTBot 4.51%, ClaudeBot 0.80%, DeepseekBot 0.02%. Zero cases of AI bots requesting a non-existent llms.txt. Google states its search systems don’t read llms.txt. AI search crawlers fetch HTML directly, so the study concludes investment belongs in HTML readability, structure and authority. Limitation: Ahrefs is an SEO tool vendor with an interest in the “classic SEO works” conclusion, and its measurement skews toward its own users’ sites, so the publisher itself notes the 28% adoption figure is an upper bound.)
- Originality.ai 12-month tracking (3M+ sites, June 2025 to May 2026). Coverage: PPC Land, “llms.txt adoption rises 8.8x but 97% of files get zero AI requests”, June 2026, ppc.land (Sites with llms.txt grew from 4,088 in June 2025 to 36,120 in May 2026, 8.8x. llms-full.txt went from 23 to 2,463 sites (107.1x) and ai.txt from 4 to 397 (99.3x). The three formats total 38,980 sites, with llms.txt accounting for 92.7% of implementations.)
