New Report: 1 in 10 Publishers in Indonesia Use Robots.txt to Block AI Crawlers
A new report, The Protocol Gap: Indonesia, examines how news publishers in the country use robots.txt to direct which AI crawlers can access their content. It finds that while 74.6% have a robots.txt file, only 10.1% use it to block at least one AI crawler as of September 2026.
AI companies are increasing their demand for data to train large language models, AI assistants, and AI-powered search tools. This puts growing pressure on news publishers to decide whether, and how, AI crawlers can access their work. One of the few tools available to publishers is robots.txt: a simple file that enables publishers to signal whether specific search engines and web crawlers are allowed to access their content.
The latest analysis, carried out on September 30, 2026, started with a sample of 1,862 news site URLs. Of these, 1,565 sites returned a result (the other 297 sites were undetermined, and we therefore could not reach a robots.txt file). Among the 1,565 sites analyzed, only 10.1% used the file to block at least one AI crawler. While this percentage is relatively low, this figure has nearly doubled from 5.8% in July 2025, suggesting awareness is slowly growing.
How Indonesian publishers implement AI crawling policies varies widely, reflecting differences in institutional resources, technical capacity, and business priorities. At the same time, publishers must navigate a complex and fast-changing landscape driven by the emergence of AI-powered search and summaries. Many publishers in Indonesia are dealing with sharp drops in traffic referrals and weakening programmatic advertising revenues. This leaves them with difficult strategic decisions and trade-offs: stay open for visibility and referrals, or restrict bots to protect content and strengthen bargaining power. Smaller outlets are especially vulnerable, because they often lack the capacity to measure their losses, assess how AI crawling affects their traffic and revenue, or engage platforms in meaningful negotiations.
Closing these gaps will require further research, collective tools, and legal reform to give publishers stronger grounds to decide how their content is used by AI platforms. Without effective ways to manage AI crawlers, publishers risk losing control over how their journalism is accessed and reused, while facing declining referral traffic, limited bargaining power, and uncertainty over compensation.
The Protocol Gap: Indonesia was produced in collaboration with the Journalism Relay Project, Asosiasi Media Siber Indonesia (AMSI), Monash University Indonesia, PR2Media, and the International Fund for Public Interest Media (IFPIM).
It is the third in a series of country reports, the first two focusing on Brazil and South Africa. The series is part of a broader cross-border research collaboration with partners in Brazil, South Africa, and Indonesia, aiming to strengthen global understanding of evolving norms and practices around AI access to journalistic content, with a particular focus on key markets in the Global South.
āI never thought Iād be automating myself out of a job. But here we are.ā
