What is llms.txt, and do you need one?

What goes in the file

The format comes from llmstxt.org. A title, a one-line summary, then sections of links with a note on each:

Only the title is required. A section called Optional holds links a tool can skip when it is short on room. The current version of the proposal also lets a file sit at any path and cover the pages beneath it, and suggests offering a plain Markdown copy of each page.

Where it came from

Jeremy Howard published the proposal on 3 September 2024. His reasoning was that a web page is mostly navigation, scripts and styling, and a language model with limited room does better with a short, clean list of what matters.

Who reads it

Mostly tools working for one person: a coding agent told to use an API, or someone who pastes an address into an assistant and asks about the product. That is why the files in the wild are concentrated on software and developer sites. The llmstxt.org page notes that OpenAI and Anthropic publish one for their own developer docs.

That is different from an assistant consulting your llms.txt when it decides which businesses to recommend. We looked for a statement from OpenAI, Anthropic or Perplexity that their assistants do this and found none. Google's guidance on its AI features says you "don't need to create new machine readable files, AI text files, or markup" to appear in them.

Chrome's Lighthouse now has an llms.txt audit, among its checks for AI agents browsing a page. It does not ask you to have one. It flags a site only when requesting the file causes a server error, and treats a missing file as not applicable, because the file is optional.

How many sites have one

On 14 November 2026 we requested llms.txt from 60 well-known sites. Twelve refused the request or did not answer. Of the 48 that gave a clear answer, 19 have a file.

So it is a habit of companies whose customers are developers, and almost nobody else's. The full study lists every site.

Ahrefs ran a much larger check in May 2026 and published it on 15 June. Of 137,210 domains with traffic, 28% had a valid llms.txt. Ahrefs says its sample leans towards technical, SEO-aware sites, so 28% is an upper bound for the web as a whole.

Do the files follow the format?

Mostly not. We ran the 19 files through our checker: 5 follow the published format on every point (Shopify, Target, Vercel, Cloudflare and Netlify). Nine list more than 100 links, and Salesforce's runs to 781 KB and 2,857 links with no title line. Etsy's and Mailchimp's are written as briefings, with links inside sentences rather than as a list.

Nobody enforces the format, and some of the files that break it may work better for their purpose than a tidy list would. The large ones have lost the point, though: the proposal exists so a tool with limited room can read something shorter than the site.

Does anyone fetch it?

Rarely. Two measurements, from two companies that sell AI visibility tools:

• Ahrefs, across many sites. Of roughly 38,000 valid files in its May 2026 sample, 97% received no requests at all that month, by its own bot analytics. Ahrefs adds that a fetch does not mean the file was read or used.
• Otterly, on one site. It put a file on a test site and watched the logs for 90 days. Of more than 62,100 visits from AI crawlers, 84 went to the file, about 0.1%. An ordinary page on the same site got around 265.

Neither shows harm, and neither shows benefit for being cited.

What we use ours for

We have one, and it is not a list of pages. People paste "set up Content Autopilot" into coding agents, so our llms.txt tells an agent how to connect to an account and what each tool does. For that job it works: the agent reads one file and knows what to do. We also publish a plain list for every blog we host, because it costs nothing.

So our view is narrow. If AI agents need to operate your product or read your documentation, llms.txt is worth doing properly. If you hope it will get a shop or a service business recommended, nothing published so far says it will.

How to write one

1. List the pages a new colleague would need on day one. What you sell, what it costs, how to get help, the questions customers ask most. Ten to forty pages is plenty.
2. Write the summary line. One sentence: what the business is and who it is for. A tool short on room may read only this.
3. Group the links under three to six headings. Products, Pricing, Guides, Help, Company. Put what an assistant most needs to get right near the top.
4. Give each link a note. A few words on what is on the page: "prices for each service level", not "learn more".
5. Move the rest under Optional. History, press, careers.
6. Save it as llms.txt at the root of the site and check it loads at yoursite.com/llms.txt as plain text.
7. Run it through a checker, then add a reminder to update it when pages move. A file that lists deleted pages is worse than no file.

A template for a service business, to copy and fill in:

# [Business name]

> [What you do] for [who], in [where]. [One fact that sets you apart.]

## Services

- [Service one](https://example.com/services/one): What is included and what it costs
- [Service two](https://example.com/services/two): What is included and what it costs

## Before you book

- [Prices](https://example.com/prices): Every price, including call-out and extras
- [Areas we cover](https://example.com/areas): Towns and postcodes
- [Questions customers ask](https://example.com/faq): Guarantees, timing, payment

## Optional

- [About us](https://example.com/about)
- [Reviews](https://example.com/reviews)

Or let the generator draft one from your sitemap, and trim it by hand.

What to do first

In the order they matter: check that the AI search crawlers are allowed in your robots.txt, check that your text is in the HTML without JavaScript, and write pages that answer what buyers ask. Then, with ten minutes left, make the file.