Ahrefs studied 137,000 domains and found almost no llms.txt files get requested. The files that do get read aren't being read by AI search bots.
Ahrefs pulled server logs and live bot traffic for 137,000 domains and found that 28% of them publish an llms.txt file. Of those roughly 38,000 files, 97% received zero requests in May 2026. No bots, no humans, nothing. That headline number has already made the rounds. The more useful finding is buried a few sections down: of the 3% of files that did get fetched, the bots doing the fetching mostly aren't the ones people are building the file for.
What the study measured
Ahrefs checked every domain root in its Web Analytics customer base for a valid llms.txt (confirming it was real markdown, not a soft 404), then used Bot Analytics to classify every request to those files by user agent. The population skews technical, since Ahrefs customers are more SEO-aware than the average site owner, so the 28% adoption figure is likely a ceiling, not a typical rate. Even so, roughly 1,100 domains accounted for every fetch the study recorded, out of 38,000 files published.
Google gave conflicting signals on this in the same month the study ran. Its own guide on optimizing for generative AI features carries a section literally titled "mythbusting" that says machine-readable files like llms.txt aren't needed to appear in generative AI search. Days later, Chrome shipped an llms.txt check inside Lighthouse's experimental agentic-browsing audits. When SEO consultant Lily Ray asked Google's John Mueller about the contradiction, he called llms.txt "not done for search," describing it instead as a "temporary crutch" for AI coding tools parsing developer documentation, not something most sites need to worry about.
Who fetches the files that get read
This is the part the 97% headline skips past. Ahrefs classified every bot hitting the 3% of files that received traffic, and broke the results into twelve categories. SEO audit tools, the crawlers behind tools like Site Audit and WebPageTest, sent 21.7% of requests. Unidentified bots sent 14.9%. General web crawlers like Googlebot sent 13.1%. Tech-profiling tools sent 11.6%. None of those are AI systems at all.
Grouped together, the four AI-related categories, agents, training crawlers, assistants, and retrieval bots, account for 19.5% of requests. That sounds respectable until you split it apart. AI agents and the infrastructure serving them (things like Claude-Code) made up 10.5%, more than any other AI category. Training crawlers, led by GPTBot, made up 5.3%. AI assistants made up 2.5%. AI retrieval bots, the category that answers a live query inside ChatGPT, Perplexity, or an AI Overview, the use case most people are picturing when they publish an llms.txt, made up just 1.1% of requests. Slackbot, generating link previews when someone shares a URL in chat, fetched llms.txt files more often than PerplexityBot did.
A separate 12% of requests came from an industry studying the file rather than using it: GEO and AEO tools scoring sites for AI readiness, dedicated llms.txt scanners and directories, and research bots, including one identifying itself as prompt-injection-survey, systematically checking whether AI agents can be tricked through the file. That's a near match for the AI bot share, meaning almost as much of the measured traffic comes from tools auditing llms.txt as from AI systems reading it for any purpose at all.
The file isn't a discovery mechanism
Ahrefs also checked every 404 on an llms.txt path, meaning a file that doesn't exist. The split was almost the inverse of the valid-file traffic: 98% human, and zero requests from identifiable AI bots. AI systems never go looking for a file that isn't there. They fetch llms.txt only when a link, an index, or an explicit instruction points them to it. A site without one isn't skipping a step that AI search would otherwise have found on its own, and a site with one isn't guaranteed a single visit from anything beyond the SEO tools now scanning for it by default.
Where this connects to actual AI visibility
This blog already covered why 88% of ChatGPT's citations still come from the same general search index that powers classic organic rankings, not from a specialized file format. The llms.txt data lines up with that finding rather than contradicting it: retrieval bots, the piece of the AI stack that decides what gets cited in an answer, are the smallest slice of llms.txt readership in this study. If a page is going to get pulled into an AI-generated answer, the mechanism doing that work is overwhelmingly the same crawling and ranking pipeline that's always mattered, not a markdown index sitting at the domain root.
That's the same pattern this blog just wrote up in the Ahrefs correlation study on YouTube mentions and backlinks: a genuinely new AI-search metric gets measured, a narrow and specific finding comes out of it, and by the time it reaches most site owners it's been flattened into a much bigger claim than the data supports. Here, "AI search is reshaping how content gets discovered" turns into "publish this file," when the file's own usage data shows AI retrieval systems barely touch it.
Should you bother
Ahrefs' own verdict, and the logic behind it, holds up: publishing an llms.txt is cheap, and some website builders already generate one automatically, so there's little cost to having one. But if the goal is AI search visibility specifically, the base rate is brutal. Check your own server logs before spending more time on this than a few minutes. If you're already investing in the things that build durable AI citation likelihood, content that ranks, gets referenced, and earns real backlinks, an llms.txt file isn't going to substitute for that work or meaningfully accelerate it. It's mostly there for the coding agents and the audit tools, at least for now.
