AI Scraping Bots Put Academic Library Access at Risk

AI scraping bots are disrupting academic library access, but support for a shared robots.txt signal is not the same as a solution. The reported problem is not simply that AI companies collect material; it is that some bots make public collections difficult or impossible to reach while doing so.

AI scraping bots turn collection access into an infrastructure issue

Academic libraries and cultural institutions are reporting bots that send excessive requests, slowing websites or taking them offline as they gather material for large language models. Some bots ignore robots.txt instructions and can request thousands of documents a second. Blocking their IP addresses is less effective when operators distribute requests across many addresses.

That changes the practical question. A collection can be legally and intentionally open while its website is still unavailable to readers. Project Gutenberg, for example, welcomes its public-domain books being used for AI training, but says bots should access them without taking the site down. The distinction matters: openness to reuse is not consent to a level of automated demand that interrupts ordinary access.

The examples show disruption, not a universal measure of scale

The source article describes more than 11,000 searches for Finnish music on one day, coming from thousands of different internet addresses, and reports bots requesting thousands of documents a second. Those examples make the operational burden concrete, but they do not establish how often this happens across all institutions or how much traffic any one bot operator generates over time.

That limit should temper broad conclusions without minimizing the problem. A small number of extreme request patterns can still disrupt a public resource; the figures are evidence of a real access failure, not a complete census of AI scraping. The relevant measure for a library is not only how much material gets copied, but whether people can still use the collection while copying is underway.

A robots.txt opt-out signal leaves enforcement unresolved

Anthropic, OpenAI, Google and Microsoft have expressed support for a standardized robots.txt opt-out signal. But the signal would be voluntary, and the proposal excludes some third-party and smaller-company bots. It therefore offers a way for participating operators to recognize a preference, not a general mechanism to ensure that every scraper respects it.

The gap matters because the hardest cases described here are bots that ignore instructions and spread requests across addresses. Meanwhile, CAPTCHA and login requirements can block helpful bots or create barriers for people. Institutions face a constrained choice: allow broad access and risk disruption, or add controls that may make legitimate use harder.

The narrower claim that survives is that standardized opt-out signals could help where operators choose to honor them; they do not, on the evidence described, guarantee reliable access to library collections. Until coverage and compliance extend beyond willing major firms, the burden of keeping open collections usable remains with the institutions hosting them.