llms.txt and AI bots: 97% of files got no fetches; coding agents fetch it, not by habit

llms.txt was proposed as a map of your site for AI systems: one small file listing what matters, written for machines. The idea is reasonable. The measured reality, as of 2026, is that AI search barely touches it.

The best data comes from Ahrefs, who checked server logs across 137,210 domains for May 2026: 97% of published llms.txt files were never fetched at all, and 77% of the bots that did fetch one were not AI tools: SEO auditors, technology profilers and ordinary crawlers. The bots AI search sends to answer questions made 1.1% of the requests. Google's John Mueller said in June 2025 that no AI system used the file, and in May 2026 called it a temporary crutch for coding tools, not something done for search. Google's own documentation asks for nothing special: AI features read the same pages as classic search.

There is one real exception, with a condition: coding agents. In Ahrefs' logs, Claude Code fetched llms.txt more often than any AI search or assistant bot, and more than every AI training crawler but GPTBot. But agents read it when something points them to it, not by habit. Ahrefs saw no AI bot ask for an llms.txt that does not exist, and in a controlled test, none of 15 coding tools and assistants, each given a documentation site's address and asked to get its docs, requested its llms.txt (arXiv 2604.02544). If developers are part of your audience, the file has a genuine, narrow job, as long as you link to it where agents will see the link.

So the honest guidance is both-ways: publish llms.txt and link to it — it costs minutes and serves the agents that are sent to it — but build your discoverability the boring way that measurably works: being in the classic search indexes with complete, server-rendered HTML. That is what this site does, every page here names its index at the top, and this page is itself part of the experiment.