llms.txt: What It Is, How the Format Works and Whether You Need One
llms.txt is a Markdown file at the root of a site that tells a language model what the site is and which pages are worth reading. It was proposed in September 2024. It is not a web standard, no major search engine says it uses it, and about one in six of the busiest sites serve one. Here is the format, the measured numbers, and a plain answer on whether to publish one.
What Is llms.txt?
llms.txt is a plain Markdown file served at /llms.txt. It introduces the site in a sentence or two, then lists the URLs a language model should read, grouped under headings. The idea is to hand an AI assistant a short, clean map instead of a whole HTML site full of menus, scripts and ads.
Jeremy Howard proposed it on llmstxt.org on September 3, 2024. It is a community proposal: no standards body publishes it, and no AI company has committed to reading it.
| Fact | Value |
|---|---|
| Proposed | September 3, 2024 |
| Spec | llmstxt.org |
| Location | /llms.txt, or a subpath |
| Format | Markdown |
| Required part | One H1 |
| Standards body | – |
| Used by Google Search | – |
It is often confused with robots.txt. They do different jobs: robots.txt says what a crawler may fetch, and crawlers follow it under RFC 9309. llms.txt only suggests what is worth reading. It allows and blocks nothing. For the crawler side, see the AI crawlers guide and the robots.txt tester guide.
The llms.txt Format, Line by Line
The spec on llmstxt.org defines the parts in a fixed order. Only the first heading is required.
| Part | Required | Markdown |
|---|---|---|
| Site or project name | ✓ | H1 (#) |
| Short summary | – | Blockquote (>) |
| Details | – | Paragraphs or lists |
| File lists | – | H2 (##) + links |
| Optional section | – | H2 named Optional |
Each item in a file list is a Markdown link, optionally followed by a colon and a note. Links under an H2 called Optional are ones an assistant may skip when it needs a shorter context. A minimal file looks like this:
# Example Store
> Hand-made ceramic tableware, shipped from Portugal to the EU and the US.
Prices are in euros. Every product page lists dimensions and care.
## Products
- [Plates](https://example.com/plates/): dinner, side and serving plates
- [Mugs](https://example.com/mugs/): 250 ml and 350 ml
## Help
- [Shipping](https://example.com/shipping/): costs and delivery times
- [Returns](https://example.com/returns/): 30 days
## Optional
- [Our story](https://example.com/about/)
The spec also suggests a Markdown copy of each useful page at the same URL with .md added, for example page.html.md. A companion file, llms-full.txt, is a common extension: the full text of the pages in one file rather than links to them. The proposal allows a file at a subpath too, such as /docs/llms.txt, covering the pages under it.
Who Actually Serves an llms.txt? The Numbers
SEO ProCheck requested five agent files from 184 of the most visited sites on September 29, 2026. Totals only:
| File | Sites serving it | Share |
|---|---|---|
| /llms.txt | 31 of 184 | 16.8% |
| /llms-full.txt | 6 of 184 | 3.3% |
| /AGENTS.md | 1 of 184 | 0.5% |
| /.well-known/llms.txt | 0 of 184 | 0% |
| /ai.txt | 0 of 184 | 0% |
Almost every site that served one sells software or developer tools. The median file was 12,490 bytes, so these are real files, not placeholders. The full method is in Who Actually Serves an llms.txt? 184 Top Sites Measured. Adoption says who is willing to publish one. It says nothing about whether any assistant reads it.
Our own two sites, checked November 21, 2026:
| Site | /llms.txt | Content-Type | Size |
|---|---|---|---|
| seoprocheck.com | 200 | text/plain | 3,078 bytes |
| neova.app | 404 | text/html | none |
SEO ProCheck publishes one because its readers are SEO people who test such files. neova.app does not, yet. Both are fine. A missing llms.txt is not an error.
Does Google or Any AI Assistant Use llms.txt?
Here is what each company says in its own documentation, as of November 21, 2026:
| Company | Reads llms.txt | Source |
|---|---|---|
| Google Search | – | AI optimization guide |
| OpenAI | Not documented | Crawler docs |
| Anthropic | Not documented | Crawler docs |
| Perplexity | Not documented | Bot docs |
Google is the only one that says it outright. Its guide to optimizing for generative AI features, published May 15, 2026, says you do not need machine readable files, AI text files, markup or Markdown to appear in Google Search. It adds that an llms.txt made for other services neither helps nor harms visibility in Google Search, because Google Search ignores it. Google's John Mueller had said in June 2025 that no AI system used the file and that server logs showed it.
Where the file can help is at inference time: a coding assistant or agent that a person points at your documentation may fetch it and follow its links. That is why developer tool companies lead adoption. llms.txt Explained on SEO ProCheck goes through the evidence, including what server logs show.
Should You Publish an llms.txt?
| Your site | Publish one? | Why |
|---|---|---|
| Developer docs or an API | ✓ | Agents read docs on request |
| Software product | ◐ | Cheap, low upside |
| Online store | ◐ | Feeds and pages matter more |
| News or publisher | – | No documented reader |
| Local business | – | Business Profile matters more |
If you publish one, it costs an hour and a little upkeep. If you do not, you lose nothing measurable in Google Search, which is the one place a company has said so in writing. What does matter for every AI assistant is that its crawler can fetch your pages and gets the main content in the HTML. Fix that first; it is covered in AI search optimization and ChatGPT SEO.
Two cautions. First, an llms.txt is public, so anything you list is a suggestion to everyone, including scrapers. Second, it is not an access control: a page left out of llms.txt is still open to any crawler robots.txt allows.
How to Write and Serve an llms.txt
- Pick the pages. 10 to 50 URLs that answer what people ask about you: product, pricing, docs, policies, contact. Not every URL; that is the sitemap's job.
- Write the H1 and the summary. The name, then one sentence a stranger would understand.
- Group the links under H2s and add a short note after each one.
- Serve it as text.
/llms.txt, status 200,Content-Type: text/plainortext/markdown, UTF-8. - Link only to live URLs. Every link should answer 200, not a redirect or a 404.
- Keep it current. Regenerate it when pages move. A stale file points assistants at dead pages.
| Item | Good | Problem |
|---|---|---|
| Status | 200 | 5xx |
| Content-Type | text/plain | text/html |
| First line | # Name | Anything else |
| Links | Answer 200 | 404 or redirect |
| Size | A few KB | Whole site inlined |
A common mistake is a CMS or CDN that answers /llms.txt with the home page or a styled 404 page and status 200. To a reader, that is an HTML page, not an llms.txt.
Check Your llms.txt and Agent Files With Neova
Neova checks a published llms.txt and the other agent files. It does not write one for you, and a site without them is not flagged. The checks:
- llms.txt Is Malformed or Returns a Server Error: a 5xx, or a 200 with a non-text Content-Type or a first line that is not a
# title. A 404, 410 or HTML page counts as no file. - Markdown Version Is Empty or Holds Under Half of the Main Text, when the start URL answers
Accept: text/markdownwith Markdown. - Agent Discovery Manifest Is Invalid or Not Served, for agent discovery files advertised in robots.txt or in a link.
- Request With an AI Search Crawler User Agent Is Blocked, because a file nobody can fetch helps nobody.
- Key HTML Page Restricts Snippets or Blocks AI Search Crawlers.
- Main Content Starts After the First 5,000 Characters of Body Text, the HTML side of the same question.
What Each AI Assistant Is Served asks sampled pages as 10 AI agents and shows who gets the full page and what robots.txt allows. These run in the desktop app on Mac, Windows and Linux and in Neova Online in the browser. The Free plan crawls 1,000 URLs with every check; see Pricing or download Neova.
Questions
What is llms.txt?
A Markdown file at /llms.txt that names a site, summarizes it and lists the pages a language model should read. It was proposed on llmstxt.org on September 3, 2024, and is not an official web standard.
Does Google use llms.txt?
No. Google's guide to generative AI features, published May 15, 2026, says Google Search ignores llms.txt and that it neither helps nor harms visibility there.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may fetch and is a standard, RFC 9309. llms.txt only suggests what to read and blocks nothing.
How many sites have an llms.txt?
In SEO ProCheck's September 29, 2026 measurement, 31 of 184 top sites served /llms.txt (16.8%), nearly all of them software or developer tool companies.
What is llms-full.txt?
A companion file that holds the full text of the listed pages in one Markdown file instead of links to them. In the same measurement, 6 of 184 sites served one.