Cloudflare is implementing measures to block known AI web crawlers by default to protect content creators from unauthorized access and scraping. New domain owners will be asked if they want to allow AI scrapers and can participate in a “Pay Per Crawl” fee program, allowing publishers to set prices for content access. This initiative is currently available for select publishers to ensure AI companies use content legally and compensatively. Previously, Cloudflare allowed selective blocking of AI crawlers based on robots.txt compliance, but now the blocking applies universally to all AI bots by default for new customers. Major publishers like The Associated Press and Quora support these restrictions as online content faces challenges from AI-driven discovery. Cloudflare aims to empower creators while collaborating with AI companies to define crawler purposes and enhance responsible content use. CEO Matthew Prince emphasized the importance of protecting original content as integral to the Internet’s value.
Source link
