On August 20, 2026, we found out our own site was turning AI crawlers away. Nobody complained. A tool we run fetched jiashley.com's robots.txt from the outside, the way a crawler does, and came back with WARN: three AI bots blocked. The file on our server said the opposite: AI crawlers welcome. What the internet was getting was a different file, one that disallowed GPTBot, ClaudeBot, Google-Extended, CCBot, Amazonbot, and Applebot-Extended, because the zone sat behind Cloudflare with two settings switched on: AI bots protection set to block, and Cloudflare's managed robots.txt. Nobody here had turned either one on deliberately. One API call to the zone's bot_management endpoint turned both off, and the next audit read PASS, none blocked. The tool said three. The file it fetched named six. As best we can tell, the tool only watches for three.1

The next day the same tool ran against the site of an agency that sells AI search optimization. GPTBot, ClaudeBot, and Google-Extended, all three disallowed in the file it serves.1 We aren't naming them. We're writing this because we didn't know about ours either until something went and looked.

You wrote a robots.txt. It's in the repo, it allows the crawlers you want, and you've probably looked at it more than once this year. Here's the thing you may not have done: fetched it from outside, the way a crawler does, and read what came back.

If your site is behind Cloudflare, that fetch can return a different file. Lyra, our research agent, went looking for how, and everything she could find came from Cloudflare's own documentation. That's worth saying up front. This is Cloudflare describing Cloudflare, so read it as the vendor's account of its own feature, and check your own site rather than taking either of us on faith.

What the managed file does

Cloudflare offers a feature it calls managed robots.txt. When it's on, Cloudflare checks whether your server already serves a robots.txt, judged by an HTTP 200 response, and if it does, Cloudflare prepends its own directives ahead of yours and sends both back as a single file.2 If your server has no robots.txt, Cloudflare creates one.3

The result is a file with two authors. Your directives are still in there, untouched. Cloudflare's sit above them. What Cloudflare's block says, in Cloudflare's own blog post about the feature, is a request to Google-Extended and Applebot-Extended, "amongst others," that they not crawl the site for AI training.4 That "amongst others" is as far as the documentation goes. Lyra couldn't find a published list of every user agent in the managed block. Our own served file on August 20 named six, GPTBot and ClaudeBot among them, while the file on our server said AI crawlers were welcome. So on our zone, at least, the block ran longer than the two names Cloudflare gives.1

One more thing the documentation is plain about: robots.txt is a request. Cloudflare's own page says compliance is voluntary and that the file does not stop a crawler at a technical level.2 So the managed file, on its own, changes what you're asking for. It doesn't change what's possible.

The setting that does block

That's the second feature. Cloudflare calls it AI Crawl Control, and its docs describe it as the way to enforce crawl blocking rather than request it.3 It blocks at the network edge, and nothing your robots.txt says reaches it. Our own record puts it about as plainly as it can be put: the origin said welcome, the edge said no, and the edge wins.1 Cloudflare suggests running the two together, robots.txt to state a preference and AI Crawl Control to enforce it.3

If you meant to allow AI crawlers, this is the one to check first. A crawler can ignore a text file. It can't ignore a block at the edge.

How to see what's actually being served

From a machine that isn't your server, request your robots.txt the way any crawler would. A browser works; so does curl. Read the whole thing, top to bottom. If there's a block near the top you didn't write, the managed file is on. Then compare that against the file on your origin. If the two differ, you've found the gap.

The network-level block won't show up in that file at all. That's the point of it. For that one you have to go into the dashboard.

Where the switches are

For the managed file, Cloudflare's instructions go to the Security Settings page, filter by Bot traffic, and find the setting labeled "Set your preference to block training in robots.txt."2 That's the switch. On prepends the block. Off stops it. We didn't go through the dashboard ourselves. One PUT to the zone's bot_management endpoint turned both off, ai_bots_protection set to disabled and is_robots_txt_managed set to false. The exact request is in the evidence file.1

About the word "silently"

The assignment that produced this piece assumed these features arrive switched on. Lyra couldn't confirm that. Cloudflare's blog says "once enabled," which reads as something a person turns on, and she found no source saying either feature is on by default on any plan.4 Our own case doesn't settle it. Nobody here turned them on deliberately; that much we know. How they ended up on, we don't. So I'm not going to tell you Cloudflare did this to you. What I can tell you is that it's a switch in a security menu, and its label describes a preference you could flip while thinking about something else entirely.

Whether that happened on your account is a quick check. The file on your server is your intent. The file the internet receives is your policy. Go read the second one.

This check is one of the things our AI visibility audit runs from the outside, along with the rest of what a crawler sees that you don't. If you'd rather not go through the dashboard yourself, that's the page.

Sources

  1. The firm's own record for this piece: the 2026-08-20 finding on jiashley.com (key finding_2026_08_20: zone settings before and after, the served file's disallow list, the fix request and body), the audit tool's robots.txt checks (key audit_robots_checks, including the 2026-08-21 entry for the unnamed agency), and a 2026-09-05 live capture. The pre-fix audit JSON was not kept; the WARN result is as recorded in the finding. cdn-blocking-ai-crawlers-2026-08-20-evidence.json. ↩
  2. Cloudflare, managed robots.txt documentation, https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/, published 2026-08-03, retrieved 2026-09-05. Quoted: "If your website already has a robots.txt file — verified by an HTTP 200 response — Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single"; "robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level"; "In the Cloudflare dashboard, go to the Security Settings page. Go to Settings ↗ Filter by Bot traffic. Go to Set your preference to block training in robots.txt." ↩
  3. cloudflare, cloudflare-docs source file for the managed robots.txt page, https://github.com/cloudflare/cloudflare-docs/blob/production/src/content/docs/bots/additional-configurations/managed-robots-txt.mdx, published unknown, retrieved 2026-09-05. Quoted: "Cloudflare detects whether your origin server already has a robots.txt file and adjusts accordingly — either merging with your existing file or creating one from"; "If you want to enforce crawl blocking rather than request it, use AI Crawl Control. You can also use both features together — robots.txt to express your preferences and AI Crawl Control to enforce them." ↩
  4. Cloudflare, "Control content use for AI training with Cloudflare's managed robots.txt", https://blog.cloudflare.com/control-content-use-for-ai-training/, published unknown, retrieved 2026-09-05. Quoted: "Cloudflare's managed robots.txt signals your preference to Google-Extended and Applebot-Extended, amongst others, that they should not crawl your site for AI"; "Once enabled, Cloudflare will automatically update your existing robots.txt or create a robots.txt file on your site". ↩

Updated September 5, 2026. Rewritten by the desk with every claim sourced and every citation verified; the original ran on the date above.