New Report: 30% of South African Publishers Block AI Crawlers via Robots.txt
The Protocol Gap: South Africa examines how South African news publishers are using robots.txt to manage AI crawler access. Based on an analysis of 263 news websites, the study finds that while 74.1% of publishers have a robots.txt file in place, only 30.4% block at least one AI crawler as of April 2026.
As generative AI companies increase their demand for data to train large language models, AI assistants and AI-powered search tools, news publishers face growing pressure to decide whether and how their content can be accessed by automated crawlers. One of the few tools available to publishers is robots.txt, a simple protocol that allows websites to signal whether specific bots should be permitted to access their content.
Although the percentage of publishers that use robots.txt to block AI crawlers rose modestly from 29.7% in December 2025 to 30.4%, the findings suggest that most publishers remain open to AI data collection.
The report also indicates that AI-blocking measures are concentrated among larger and better-resourced publishers, while many smaller, independent and community outlets remain exposed to scraping. This raises important questions about bargaining power, content ownership and sustainability, as AI systems become increasingly reliant on news content.
Produced by the Media Leadership Think Tank (GIBS), Journalism Relay Project, and the International Fund for Public Interest Media, this report is part of a broader research collaboration examining AI scraping of news content in Global South markets such as South Africa, Brazil, and Indonesia. A first study published in December 2025, The Protocol Gap: Brazil, found that only 7.2% of Brazilian news sites blocked at least one “AI crawler” through their robots.txt file.
In the absence of clear legal and regulatory frameworks, the decisions publishers make now about how AI crawlers access their content could have long-lasting consequences. The report’s findings contribute to ongoing debates on South Africa's Media and Digital Platforms Market Inquiry, copyright reform, and the development of a national AI policy framework.
The study also provides a replicable methodology for researchers and publisher associations interested in conducting similar audits and concludes with next steps for research and engagement with ongoing efforts to strengthen publisher agency, bargaining power, and transparency in emerging AI licensing ecosystems.
“I never thought I’d be automating myself out of a job. But here we are.”
