AI crawlers
llms.txt
A Markdown file at the root of a site that lists its most useful pages for AI tools. A proposal, now in version 2, used mostly for software documentation.
The format comes from llmstxt.org: a # title, an optional one-line summary, then ## sections of links with a short note on each. Version 2, published on 10 August 2026, lets a file cover the pages under its own path and lets each page point to the file that covers it with rel="describedby".
The evidence that it gets anyone cited is thin. Ahrefs found valid files on 28% of 137,210 domains in May 2026, and 97% of those files received no requests that month. Of the requests the files did get, bots that fetch pages for AI search answers made 1.1%. Google's guide to its generative AI features, updated in July 2026, lists llms.txt among the tactics you can ignore for Google Search.
It earns its place with agents that read documentation to do a job. Of the 19 large sites we found with one on 14 November 2026, 16 are software or developer companies.