TollBit found ~15% of AI page-fetchers reached URLs European sites explicitly disallowed. ChatGPT-User, Bytespider, and Youbot bypassed it nearly half the time.
Every "block AI crawlers with robots.txt" guide runs on the same assumption: write the right Disallow line and the bot stays out. TollBit's latest State of the Bots report, covering the first half of 2026 and reported by Search Engine Journal on August 14, tested that assumption against real server data. Across the European sites in the study, roughly 15% of identified AI page-fetchers reached URLs those sites had explicitly disallowed. Three specific agents, ChatGPT-User, Bytespider, and Youbot, bypassed the file on nearly half of the sites that had named them in robots.txt.
If you've been treating your robots.txt file as a wall, the data says it's closer to a sign. Some bots read the sign and stop. Some read it and keep going.
The category that matters here
TollBit's measurement isn't about the crawlers most robots.txt guides focus on. It's not GPTBot or CCBot, the crawlers that sweep the web to build training data, and it's not OAI-SearchBot, the crawler that decides whether a site shows up in ChatGPT's search results. The bots in this study are page fetchers: agents that load one specific URL in real time because a person typed a question into a chat window and the assistant needed that page to answer it.
That distinction matters because the three categories behave differently, and most generic robots.txt guides for AI bots lump all of them under one Disallow block without flagging that the rules governing them aren't the same.
Why the file doesn't bind everyone the same way
According to OpenAI's own crawler documentation, ChatGPT-User visits a page when a ChatGPT user asks about it, and because a person initiated that request, robots.txt rules may not apply. Perplexity takes the same position for Perplexity-User. Anthropic draws a different line: it states that all three of its bots, ClaudeBot, Claude-User, and Claude-SearchBot, respect the file, a position it confirmed in February 2026.
This isn't a fringe interpretation. Google formalized the same category in March 2026 when it added Google-Agent to its official crawler list, documenting that user-triggered fetchers generally ignore robots.txt because a person requested the page. TollBit's methodology doesn't parse any of this nuance. It counts any request to a disallowed URL as a bypass, regardless of what the operator's documentation claims about intent.
The exception that isn't really an exception
Here's where the numbers get interesting. European sites barely bother blocking the newer fetch agents: only 9% disallow Claude-User, compared with 26% in North America, and Perplexity-User sits at 13% versus 26%. Most of the newest agents show single-digit disallow rates across Europe.
ChatGPT-User breaks that pattern in both directions. It's disallowed by more European sites than any comparable bot, and it also reached disallowed pages on more sites than any other bot in the study. Those two facts aren't a contradiction. An agent nobody names in robots.txt can't register a bypass of a rule that was never written. ChatGPT-User shows up more often on both sides of the ledger because it's the one agent enough sites tried to block.
The trade nobody's making on purpose
OpenAI splits its bots by function: OAI-SearchBot controls whether a site appears in ChatGPT's search results, and that instruction is one OpenAI documents as honored. ChatGPT-User handles the page fetch once a user's question points there, and that's the instruction carrying the user-initiated carve-out. A site that disallows both agents to keep AI traffic out entirely gives up the search-visibility half of that arrangement while keeping a fetching control that may not hold.
Blocking isn't free either way. Research from Rutgers Business School and The Wharton School, tracked through human browsing panel data, found publishers that blocked AI crawlers through robots.txt lost roughly 7% of weekly website traffic within six weeks. IAB Australia reached a similar conclusion from the compliance side in July 2026, issuing guidance that classifies robots.txt as unreliable for controlling live agents specifically.
What actually changes the outcome
Enforcement is starting to move below the file entirely. Cloudflare's policy update splits AI traffic into Search, Agent, and Training categories rather than a single blanket label, and starting September 15, 2026, newly onboarded domains will have Training and Agent crawlers, the exact category TollBit measured bypassing disallow lines, blocked by default on any page carrying ads. Search crawlers stay allowed by default. That's a decision made at the network edge, where the bot's stated policy on robots.txt stops being the deciding factor because the request never reaches the origin server in the first place.
This blog has already covered how a controlled crawl experiment found several major AI crawlers missing JavaScript-rendered links entirely, a different failure mode from the one TollBit measured here but the same underlying lesson: what a crawler is technically capable of doing and what it does on your specific site are two different questions, and the only way to answer the second one is to look at your own logs.
What to actually do with this
Stop treating robots.txt as an access-control system and start treating it as one input among several. Check your server logs or CDN records for ChatGPT-User, Bytespider, and Youbot specifically, since those are the three agents TollBit found reaching disallowed pages most often. If you're deciding whether to block ChatGPT-User, know that OpenAI's own documentation gives you no guarantee the block holds, and that blocking OAI-SearchBot alongside it costs you ChatGPT search visibility for certain in exchange for a fetching control that isn't. For sites that genuinely need enforcement rather than a polite request, that decision now belongs at the CDN or network layer, not in a text file sitting at the domain root.
