Content publishers traditionally welcomed crawlers and bots from search engines, as they drove traffic to their sites, allowing monetization through user clicks. However, artificial intelligence (AI) bots now operate differently, repackaging web content without necessitating visits to the original sites. This change has led to a drop in user engagement, as AI-generated citations rarely result in clicks. Cloudflare recently launched a metric on its AI Insights page, allowing site owners to see the ratio of AI crawl requests to actual user referrals, aiding them in decision-making about which AI bots to permit or block. The analysis reveals a trend where AI crawlers make significantly more requests than they generate traffic, diminishing the viability of content monetization. To counteract this shift, Cloudflare introduced tools for publishers to block unwanted AI crawlers and expanded its Verified Bots directory, providing crucial insights into bot behavior and traffic trends.
Source link
Share
Read more