New Report Reveals Gap In How Brazilian News Sites Control AI Crawlers

A new report, The Protocol Gap: Brazil, reveals that most Brazilian news websites are not taking basic steps to control how artificial intelligence companies use their content. As of November 2025, 7.2% of Brazilian news sites block at least one “AI crawler” through their robots.txt files, even though 75% have the file installed.

Robots.txt is a simple text file placed on a website’s root domain that provides instructions to web crawlers about which pages or sections they are allowed to access. While compliance is not legally binding on its own, it is one of the few free and widely recognized tools available for publishers to signal their preferences regarding AI scraping.

AI crawlers are automated programs that systematically browse and collect data from websites to train large language models and power AI assistants and search engines. The findings show that the small number of outlets restricting access focus mainly on well-known companies such as OpenAI, Google, Common Crawl, ByteDance, Amazon, Apple, Meta, and Huawei. Overall, robots.txt is rarely used as a strategic tool to signal publishers’ preferences on AI scraping.

The impact of AI crawlers on digital media business models was one of the key issues highlighted during CTRL+J conferences in 2025. Participants advocated for technical defense mechanisms to strengthen publisher controls and enable publishers to monetize bot access based on granular permissions.

‍

Produced by the Journalism Relay Project, Momentum - Journalism and Tech Task Force, and the International Fund for Public Interest Media, this first report is part of a broader research collaboration examining AI access to news content in Global South markets such as Brazil, Indonesia, and South Africa.

The report includes a detailed methodology for all those who may want to replicate this research on robots.txt in other countries and regions.

Read the full report in English, Portuguese and Spanish.