AI Scraping Bots: Revolutionizing Access to Libraries, Archives, and Museums

A recent survey highlights the disruptive impact of AI bots on libraries, archives, museums, and galleries. These bots, which scrape data for AI training, have overwhelmed the servers of numerous institutions, sometimes rendering valuable collections inaccessible. The report, authored by Michael Weinberg of the GLAM-E Lab, surveyed 43 institutions across Europe, North America, and Oceania, revealing that 39 experienced increased traffic attributed to these bots. Respondents likened the situation to Distributed Denial of Service (DDoS) attacks, as bots frequently disrupt access to resources. Despite efforts to mitigate bot-related issues, such as deploying firewalls, many institutions lack the resources or staff to combat the problem effectively. Additionally, the commonly used robots.txt protocol to block bots is often disregarded. The report indicates that while the problem is widespread, not all institutions are aware of bot scraping until their server performance declines. Overall, the situation threatens the sustainability of open-access cultural resources.

Source link