Cloudflare Starts Blocking Some Search Crawlers on September 15. Check Your Site This Week
Cloudflare flips its AI crawler defaults on September 15, 2026, and the change can block Googlebot from sites that block AI training crawlers. That includes sites that enabled the old "Block AI bots" toggle and forgot about it. If your domain sits behind Cloudflare, spend five minutes in the dashboard this week, because the failure mode here is your pages quietly vanishing from Google while the site itself keeps loading fine for everyone who already knows your address.
I build and maintain sites for small businesses, and most of them are on Cloudflare's free plan because it is free, fast and one less thing to think about. That is exactly the group the new defaults hit hardest. Cloudflare announced the policy on July 1 in a blog post it titled "Your site, your rules," and the wording there is worth reading before you decide this does not concern you.
What changes on September 15
Cloudflare is replacing its single AI-or-not bot switch with three crawler categories: Search, Agent and Training. From September 15, the defaults for each are:
- Training crawlers are blocked by default on pages that display ads, and so are Agent crawlers acting for a user.
- Search crawlers stay allowed by default.
- Crawlers that will not declare which of the three they are doing get shut out of ad-supported pages entirely.
Then comes the part that catches people. Cloudflare says multi-purpose crawlers will be assessed against every category they belong to, and the strictest rule wins. Googlebot indexes pages for search, and it also feeds Google's AI systems. So on a site that blocks Training crawlers, Googlebot is blocked, even though Search is allowed. Applebot and Bingbot sit in the same bucket, and Search Engine Watch reported that Cloudflare is removing the exemption that had kept search crawlers outside its AI blocking rules.
Three groups inherit the new defaults whether they ask for them or not: new Cloudflare customers, new sites added under an existing account, and every site still on the free tier whose owner has never touched the bot settings. Paid customers with existing sites keep their current configuration until they log in and change it.
Why a service business should care
For about thirty years the arrangement with crawlers was simple. They take your pages, they send you visitors. Cloudflare's own position is that this deal broke, and there is data behind that view. Rand Fishkin of SparkToro measured that 68% of Google searches in the first four months of 2026 ended without a single click to any website. Chartbeat data reported by Axios found small publishers lost around 60% of their search referral traffic over two years, roughly triple the drop large publishers saw.
So the referrals were thin already. Losing Googlebot is a different kind of problem. A page that Google cannot crawl stops appearing in results at all, and no amount of keyword work or link building fixes that, because the page is not in the index to begin with. And nothing on your side looks broken. Visitors who type your name still land on the site, your hosting bill stays the same, and the enquiries just get fewer over a month or two. That is a slow, quiet failure, which is why I would rather you check now than diagnose it in November.
How to check your settings in five minutes
If you use Cloudflare, this is the whole job:
- Log in, pick the domain, and open Security settings.
- Find the AI crawler or bot control section. The new options to manage Search, Agent and Training traffic are live there already, alongside the older "Block AI bots" switch.
- Look for anything blocking Training, and for anything blocking "AI bots" generally. If training is blocked, Googlebot is in the blast radius after September 15.
- Decide deliberately. Giving up AI training data does not have to mean giving up Google. Google's own robots.txt directive,
Google-Extended, lets you opt out of AI training while leaving normal crawling alone. Set that in robots.txt and keep the network-level block off. - Note the date you checked and what the setting was, so you can compare if Search Console numbers move later.
If you want a real test rather than reading settings, open Search Console, use URL inspection on a key page, and run the live test. That request comes from Google's own infrastructure, so it tells you whether Google can reach you, not whether some script pretending to be Googlebot can.
Do I need to worry if my site is not on Cloudflare?
Not about this specific change, since it only applies inside Cloudflare's network. The pattern is worth one check anywhere, though. Plenty of hosting panels, security plugins and WAF setups offer a single "block AI crawlers" switch, and some match bot names loosely enough to catch real search crawlers. If anything sits in front of your site that can answer a request with a 403, test it once from a tool that reports a live fetch as Google, and keep the result somewhere you will find it.
Is Pay Per Use worth it for a small site?
Cloudflare also renamed its Pay Per Crawl marketplace to Pay Per Use, and the idea is nicer than the old one: publishers get paid when their content actually shapes an answer instead of every time a bot fetches a page. Launch partners are Ceramic.ai and You.com. My honest read for a freelance developer's client list: this matters to people who publish content for a living. A dentist's website with eight pages is not going to earn meaningful money from citation payments, and the setup time is real. Turn it on if you are curious, do not build a business plan around it. Cloudflare says more than half of web traffic is nonhuman now, and that is the part I would actually take seriously. The traffic walking your site is increasingly not a person, so your pages need to be readable and useful to machines as well as humans.
If Googlebot does get blocked, what recovery looks like
The fix is fast once you know where to look, and the diagnosis is the slow part. Start in Cloudflare and check whether the block is coming from the dashboard rather than your robots.txt, because the two have completely different fixes. If the CDN is refusing the crawler, changing robots.txt does nothing at all, since the request never reaches your files. Turn the restriction off, then go to Search Console and use URL inspection to fetch a key page. If Google can reach it again, you will usually see the pages return to results over the following days and weeks without you submitting anything. If you want to hurry it along, submit the sitemap again and request indexing on the handful of pages that earn you enquiries, not all forty of them.
One thing to avoid: do not respond by loosening everything. Blocking training crawlers is a reasonable choice. Blocking search crawlers by accident is not, and the settings that cause it sit two rows apart in the same screen.
What the Agent category means outside search
Agent crawlers are the other half of this change, and they matter even if your site never gets blocked. Cloudflare is now treating crawlers that act on someone's behalf as a separate category from crawlers that read. That is the same shift I wrote about when AI shopping agents started showing up on small business sites. A reader takes your text and decides whether to mention you. An agent goes further and tries to complete the thing, booking the slot, filling the enquiry form, checking whether the item is in stock.
If your forms are wrapped in a challenge that assumes a person with a mouse, an agent hits a wall and moves to the next result. So the same settings screen that can lock Google out also decides whether those agents can use your booking page. I keep enquiry forms simple for this reason: labels tied to inputs, a plain submit, and a thank-you page with a real URL rather than a JavaScript popup. It costs nothing and it keeps both kinds of visitor working.
What I do on every build now
A few things carry over to every site I hand over. robots.txt and sitemap.xml stay free and crawlable, never behind a paywall rule or a bot challenge. Service pages stay as real text rather than text baked into images, which matters for search crawlers and for the AI readers I wrote about in my post on Google's Search Console AI report. The handover notes list which CDN holds the domain and which settings could cut the site off from Google, because the next person touching that dashboard should not have to reverse-engineer it.
That last point is the one owners skip, and it is the cheapest insurance here. Whoever holds your DNS is holding your visibility.
If you are on Cloudflare on the free plan and you have never opened the bot settings, you are in the group that gets the new defaults automatically. Open the dashboard, check what is blocked, and get out. If you are not on Cloudflare, nothing today, nothing next week. But if you would rather someone else check the plumbing and keep it clean, that is the work I do, you can see past builds here, and there is a short form to start a project.
The deadline is September 15. The block takes effect whether or not anyone reads the announcement.