Cloudflare Blocks AI Scrapers, Adds Pay-Per-Crawl Model

 Cloudflare will now block AI web scrapers by default and introduce a Pay-Per-Crawl system to let site owners control and monetize AI data access.

Blog News:Cloudflare, which powers about 20% of all websites, announced a major policy shift on Tuesday: AI web scrapers will now be blocked by default unless site owners explicitly grant access.

 AI Bots Restricted

Website operators can decide which AI crawlers to allow and on what terms, forcing AI companies to be transparent about whether their bots are used for training, inference, or search.

The Broken Content ModelCloudflare explained that the internet has long operated on a cycle: search engines index websites, send traffic back to creators, and generate ad revenue. AI tools, however, break this cycle by scraping text, images, and articles to generate direct answers without crediting or compensating the original creators.

“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model,” said Matthew Prince, Cloudflare’s Co-founder and CEO.

He stressed that original content is the internet’s foundation, but uncontrolled AI scraping undermines the incentive to produce high-quality work.

Pay-Per-Crawl: A New Economic ModelAlongside blocking scrapers, Cloudflare introduced Pay Per Crawl, a system allowing site owners to set micropayments for every AI bot request. This could create a marketplace where creators are compensated for their data contributions.

“Pay Per Crawl will let creators control access and get paid, ensuring AI companies can use content the right way — with permission and compensation,” said Will Allen, Cloudflare’s Head of AI Control, Privacy and Media Products.

Industry voices see promise in this approach. Jason Dion of Akylade compared it to API usage fees: “You pay for what you use, just like ChatGPT charges per token.”

Mixed Reactions from AnalystsNot everyone is convinced. Allie Mellen of Forrester warned that only highly trusted sites may benefit, while smaller publishers could stagnate. Andy Jung of TechFreedom suggested AI companies may adopt the model simply to avoid legal disputes and accusations of “pirating” content, as seen in recent lawsuits.

Still, the move represents one of the most significant shifts yet in the battle over who controls — and profits from — online content in the AI era.