Cloudflare has recently set a default policy to block AI web crawlers to protect online publishers. This decision arises from concerns that major AI firms, including Google and Microsoft, utilize web content without adequately compensating creators, undermining the original agreement where content would be indexed in exchange for traffic referral. Cloudflare’s CEO, Matthew Prince, argues that the current AI-driven landscape fails to reward content creators as the traditional search-driven web did. The disparity in traffic returned to websites can be significant; for instance, Claude’s AI made 71,000 requests for every HTML page referral. Cloudflare intends to foster fair compensation by implementing a “Pay per Crawl” service, which would require AI crawlers to pay publishers for access to their content. This shift could transform the web, potentially making it less open but crucial for incentivizing content production and maintaining quality online material.
Source link
