OpenAI is enhancing its crawling infrastructure in preparation for the holiday shopping season, particularly around Thanksgiving and Black Friday. Vercel has observed OpenAI introducing numerous new IP ranges for its bots and crawlers. Ryan Siddle from Merj highlighted this development on LinkedIn, noting the addition of several /28 blocks to OpenAI’s user IPs, which indicates a substantial increase over the last 1-2 months. He advises website administrators to integrate these new IPs into their Web Application Firewall (WAF) rules or use CDN bot detection to manage traffic effectively. The growing number of crawlers is sparking discussions in forums, emphasizing the importance of proactive measures in web security and SEO management. As OpenAI ramps up its activities, website owners should be prepared to handle the increased bot traffic to ensure optimal site performance and user experience during the peak shopping period.
Source link
