Cloudflare has enhanced its tools to combat AI scraping, now blocking AI crawlers by default for its customers and introducing a Pay Per Crawl program. This response arises as AI-driven web crawlers increasingly mimic DDoS attacks, impacting server functionality and prompting concerns from content publishers over unauthorized scraping of their material. Cloudflare has already seen over 1 million websites activate its older bot-blocking tools and is expanding this capability to millions more, utilizing a mix of behavioral analysis and machine learning to distinguish between harmful AI bots and legitimate ones. Although the Robots Exclusion Protocol (robots.txt) exists for bot regulation, many AI companies reportedly ignore it. By implementing default blocking, Cloudflare may shift the power dynamic, compelling AI firms to negotiate content access and potentially leading to compensated agreements. However, while some AI firms are onboard with this program, skepticism remains over broader industry adoption as scrapers attempt to bypass these new restrictions.
Source link
Share
Read more