Hostinger’s analysis of 66 billion bot requests across 5 million websites reveals two distinct paths for AI crawlers. Training bots, notably OpenAI’s GPTBot, are facing increased blocks, decreasing their web coverage from 84% to 12%. This surge in rejections aligns with findings that 79% of top news publishers now restrict access to training bots, with high disallow directives noted by Cloudflare. Conversely, AI assistant bots like OpenAI’s OAI-SearchBot are thriving, achieving an average coverage of 55.67%. These targeted bots directly service users and gather content without impacting visibility in the same way. Meanwhile, traditional search bot coverage remains stable, with Googlebot and Bingbot holding steady. Hostinger suggests a balanced approach—block training bots while permitting assistant bots to enhance AI visibility. For optimal results, site operators should monitor server logs and tailor blocking strategies based on their specific goals. Adopting effective SEO practices can facilitate participation in AI-driven search results.
Source link
Share
Read more