llms.txt: what it is and what it actually does

llms.txt is a markdown file at the root of your domain that tells AI agents what your site is and which pages matter. It was proposed by Jeremy Howard of Answer.AI in September 2024 and specified at llmstxt.org. It is a description, not a permission file: no crawler is required to read it, and it grants nothing. It costs ten minutes to publish and helps agents that do read it get your product right.

What goes in an llms.txt file?

The spec is deliberately small. A conformant file has an H1 with the site or product name, a blockquote summary, optional context paragraphs, then H2 sections containing markdown link lists, each link optionally followed by a short description.

> Acme turns invoices into cash flow forecasts for small agencies.

Docs

- Quickstart: connect a bank account and import invoices - API reference: REST endpoints and auth ```

The format is markdown because that is what language models read most reliably. There is no schema to validate against beyond the structure, which is why generators (including ours) mostly exist to keep the shape right.

Why does llms.txt exist?

An agent answering a question about your product has two bad options: crawl your navigation and hope, or work from whatever it remembers. Both produce confident, wrong descriptions of what you sell. llms.txt is a curated briefing that removes the guesswork: this is the product, this is the quickstart, this is the pricing page, this is the API.

It is not a sitemap. A sitemap is an inventory for search crawlers that want every URL. llms.txt is an editorial selection for a reader with a small context window and no patience. Ten to twenty links beats five hundred.

Do AI engines actually read llms.txt?

Some agents do, and no major search engine has committed to it. That is the honest state of adoption. Coding assistants and documentation agents fetch it when it exists, several developer tool vendors publish one, and the spec has an active repository. But there is no published statement from OpenAI, Anthropic or Google promising to consume the file, and you should not expect rankings to move because you added one.

So why publish it? Three reasons that survive the uncertainty:

• The cost is near zero and it is a static file, so it cannot rot into a maintenance burden.
• When an agent does read it, the payoff is direct: accurate description, correct links, fewer invented facts about your product.
• Writing it forces you to name the ten pages that matter, which is a useful exercise regardless of who reads the result.

Treat it as cheap insurance on a format that may become standard, not as an SEO tactic with a measurable return today.

Where do you put llms.txt and how do you keep it current?

Serve it at https://yourdomain.com/llms.txt as text/markdown or text/plain, on the same host you want described. Subdomains need their own file, exactly like robots.txt.

Keeping it current is the part people get wrong. A stale llms.txt is worse than none, because it feeds agents confident and outdated claims. Two habits fix it: regenerate the file when your top-level navigation changes, and link only to pages stable enough to survive a redesign. If you publish a markdown mirror of your docs, link the mirror, since it is what an agent would rather read anyway.

Frequently asked questions

Is llms.txt an official standard?

No. It is a community proposal with a public spec and growing adoption, not a standard ratified by a standards body or endorsed by the major AI vendors. Publishing it is a low-cost bet, not a requirement.

Does llms.txt replace robots.txt?

No, they solve different problems. robots.txt controls which crawlers may fetch what, and it is honored by the documented AI bots. llms.txt describes your site to agents that choose to read it. You want both, and neither substitutes for the other.

Will llms.txt improve my search rankings?

There is no evidence that it does, and no search engine has said it is a ranking signal. The realistic benefit is accuracy inside AI answers and agent workflows, not position in Google.

How long should llms.txt be?

Short enough to read in one pass, usually under a hundred lines. If you need more, the spec allows an expanded llms-full.txt alongside it, but start with the curated version.