Cloudflare’s new “pay per crawl” initiative has sparked a debate among SEO professionals and digital marketers.
The company has introduced a default AI crawler-blocking system alongside new monetization options for publishers.
This enables publishers to charge AI companies for access, which could impact how web content is consumed and valued in the age of generative search.
Cloudflare’s New Default: Block AI Crawlers
The system, now in private beta, blocks known AI crawlers by default for new Cloudflare domains.
Publishers can choose one of three access settings for each crawler:
- Allow – Grant unrestricted access
- Charge – Require payment at the configured, domain-wide price
- Block – Deny access entirely
Crawlers that attempt to access blocked content will receive a 402 Payment Required response. Publishers set a flat, sitewide price per request, and Cloudflare handles billing and revenue distribution.
Cloudflare wrote:
“Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content.
Technical Details & Publisher Adoption
The system integrates directly with Cloudflare’s bot management tools and works alongside existing WAF rules and robots.txt files. Authentication is handled using Ed25519 key pairs and HTTP message signatures to prevent spoofing.
Cloudflare says early adopters include major publishers like Condé Nast, Time, The Atlantic, AP, BuzzFeed, Reddit, Pinterest, Quora, and others.
While the current setup supports only flat pricing, the company plans to explore dynamic and granular pricing models in future iterations.
SEO Community Shares Concerns
While Cloudflare’s new controls can be changed manually, several SEO experts are concerned about the impact of making the system opt-out rather than opt-in.
“This won’t end well,” wrote Duane Forrester, Vice President of Industry Insights at Yext, warning that businesses may struggle to appear in AI-powered answers without realizing crawler access is being blocked unless a fee is paid.
Lily Ray, Vice President of SEO Strategy and Research at Amsive Digital, noted the change is likely to spark urgent conversations with clients, especially those unaware that their sites might now be invisible to AI crawlers by default.
Ryan Jones, Senior Vice President of SEO at Razorfish, expressed that most of his client sites actually want AI crawlers to access their content for visibility reasons.
Some Say It’s a Necessary Reset
Some in the community welcome the move as a long-overdue rebalancing of content economics.
“A force is needed to tilt the balance back to where it once was,” said Pedro Dias, Technical SEO Consultant and former member of Google’s Search Quality team. He suggests that the current dynamic favors AI companies at the expense of publishers.
Ilya Grigorik, Distinguished Engineer and Technical Advisor at Shopify, praised the use of cryptographic authentication, saying it’s “much needed” given how difficult it is to distinguish between legitimate and malicious bots.
Under the new system, crawlers must authenticate using public key cryptography and declare payment intent via custom HTTP headers.
Managing Crawler Access: What You Can Do
As Cloudflare’s new default settings take effect, there’s concern around losing visibility in AI search tools. But you have options to regain control.
AI traffic may decline sharply if your domain blocks AI bots without you realizing it.
Himanshu Sharma, digital analytics consultant and founder of OptimizeSmart, warned on X:
“Expect a sharp decline in AI traffic reported by GA4 as Cloudflare blocks almost all known AI crawlers/bots from scraping your website content by default.”
Sharma advised site owners to proactively review their Cloudflare settings:
- Navigate to your Cloudflare dashboard
- Go to Security > Bots
- Find the “Block AI Training Bots” setting
- Use the dropdown to change it from the default “Block all pages” to “Do not block (off)” if you want to allow AI crawlers
This option allows you to decide whether you want to maintain visibility on AI-driven platforms or limit the use of content for training and responses.
Looking Ahead
Cloudflare’s pay-per-crawl system formalizes a new layer of negotiation over who gets to access web content, and at what cost.
For SEO pros, this adds complexity: visibility may now depend not just on ranking, but on crawler access settings, payment policies, and bot authentication.
While some see this as empowering publishers, others warn it could fragment the open web, where content access varies based on infrastructure and paywalls.
If generative AI becomes a core part of how people search, and the pipes feeding that AI are now toll roads, websites will need to manage visibility across a growing patchwork of systems, policies, and financial models.
Featured Image: Roman Samborskyi/Shutterstock
