NeovaAdvanced SEO Platform

llms.txt: What It Is, How the Format Works and Whether You Need One

llms.txt is a Markdown file at the root of a site that tells a language model what the site is and which pages are worth reading. It was proposed in September 2024. It is not a web standard, no major search engine says it uses it, and about one in six of the busiest sites serve one. Here is the format, the measured numbers, and a plain answer on whether to publish one.

AI Searchllms.txtUpdated November 21, 2026By the Neova team

What Is llms.txt?

llms.txt is a plain Markdown file served at /llms.txt. It introduces the site in a sentence or two, then lists the URLs a language model should read, grouped under headings. The idea is to hand an AI assistant a short, clean map instead of a whole HTML site full of menus, scripts and ads.

Jeremy Howard proposed it on llmstxt.org on September 3, 2024. It is a community proposal: no standards body publishes it, and no AI company has committed to reading it.

FactValue
ProposedSeptember 3, 2024
Specllmstxt.org
Location/llms.txt, or a subpath
FormatMarkdown
Required partOne H1
Standards body–
Used by Google Search–

It is often confused with robots.txt. They do different jobs: robots.txt says what a crawler may fetch, and crawlers follow it under RFC 9309. llms.txt only suggests what is worth reading. It allows and blocks nothing. For the crawler side, see the AI crawlers guide and the robots.txt tester guide.

The llms.txt Format, Line by Line

The spec on llmstxt.org defines the parts in a fixed order. Only the first heading is required.

PartRequiredMarkdown
Site or project name✓H1 (#)
Short summary–Blockquote (>)
Details–Paragraphs or lists
File lists–H2 (##) + links
Optional section–H2 named Optional

Each item in a file list is a Markdown link, optionally followed by a colon and a note. Links under an H2 called Optional are ones an assistant may skip when it needs a shorter context. A minimal file looks like this:

# Example Store

> Hand-made ceramic tableware, shipped from Portugal to the EU and the US.

Prices are in euros. Every product page lists dimensions and care.

## Products
- [Plates](https://example.com/plates/): dinner, side and serving plates
- [Mugs](https://example.com/mugs/): 250 ml and 350 ml

## Help
- [Shipping](https://example.com/shipping/): costs and delivery times
- [Returns](https://example.com/returns/): 30 days

## Optional
- [Our story](https://example.com/about/)

The spec also suggests a Markdown copy of each useful page at the same URL with .md added, for example page.html.md. A companion file, llms-full.txt, is a common extension: the full text of the pages in one file rather than links to them. The proposal allows a file at a subpath too, such as /docs/llms.txt, covering the pages under it.

Who Actually Serves an llms.txt? The Numbers

SEO ProCheck requested five agent files from 184 of the most visited sites on September 29, 2026. Totals only:

FileSites serving itShare
/llms.txt31 of 18416.8%
/llms-full.txt6 of 1843.3%
/AGENTS.md1 of 1840.5%
/.well-known/llms.txt0 of 1840%
/ai.txt0 of 1840%

Almost every site that served one sells software or developer tools. The median file was 12,490 bytes, so these are real files, not placeholders. The full method is in Who Actually Serves an llms.txt? 184 Top Sites Measured. Adoption says who is willing to publish one. It says nothing about whether any assistant reads it.

Our own two sites, checked November 21, 2026:

Site/llms.txtContent-TypeSize
seoprocheck.com200text/plain3,078 bytes
neova.app404text/htmlnone

SEO ProCheck publishes one because its readers are SEO people who test such files. neova.app does not, yet. Both are fine. A missing llms.txt is not an error.

Does Google or Any AI Assistant Use llms.txt?

Here is what each company says in its own documentation, as of November 21, 2026:

CompanyReads llms.txtSource
Google Search–AI optimization guide
OpenAINot documentedCrawler docs
AnthropicNot documentedCrawler docs
PerplexityNot documentedBot docs

Google is the only one that says it outright. Its guide to optimizing for generative AI features, published May 15, 2026, says you do not need machine readable files, AI text files, markup or Markdown to appear in Google Search. It adds that an llms.txt made for other services neither helps nor harms visibility in Google Search, because Google Search ignores it. Google's John Mueller had said in June 2025 that no AI system used the file and that server logs showed it.

Where the file can help is at inference time: a coding assistant or agent that a person points at your documentation may fetch it and follow its links. That is why developer tool companies lead adoption. llms.txt Explained on SEO ProCheck goes through the evidence, including what server logs show.

Should You Publish an llms.txt?

Your sitePublish one?Why
Developer docs or an API✓Agents read docs on request
Software product◐Cheap, low upside
Online store◐Feeds and pages matter more
News or publisher–No documented reader
Local business–Business Profile matters more

If you publish one, it costs an hour and a little upkeep. If you do not, you lose nothing measurable in Google Search, which is the one place a company has said so in writing. What does matter for every AI assistant is that its crawler can fetch your pages and gets the main content in the HTML. Fix that first; it is covered in AI search optimization and ChatGPT SEO.

Two cautions. First, an llms.txt is public, so anything you list is a suggestion to everyone, including scrapers. Second, it is not an access control: a page left out of llms.txt is still open to any crawler robots.txt allows.

How to Write and Serve an llms.txt

  1. Pick the pages. 10 to 50 URLs that answer what people ask about you: product, pricing, docs, policies, contact. Not every URL; that is the sitemap's job.
  2. Write the H1 and the summary. The name, then one sentence a stranger would understand.
  3. Group the links under H2s and add a short note after each one.
  4. Serve it as text. /llms.txt, status 200, Content-Type: text/plain or text/markdown, UTF-8.
  5. Link only to live URLs. Every link should answer 200, not a redirect or a 404.
  6. Keep it current. Regenerate it when pages move. A stale file points assistants at dead pages.
ItemGoodProblem
Status2005xx
Content-Typetext/plaintext/html
First line# NameAnything else
LinksAnswer 200404 or redirect
SizeA few KBWhole site inlined

A common mistake is a CMS or CDN that answers /llms.txt with the home page or a styled 404 page and status 200. To a reader, that is an HTML page, not an llms.txt.

Check Your llms.txt and Agent Files With Neova

Neova checks a published llms.txt and the other agent files. It does not write one for you, and a site without them is not flagged. The checks:

What Each AI Assistant Is Served asks sampled pages as 10 AI agents and shows who gets the full page and what robots.txt allows. These run in the desktop app on Mac, Windows and Linux and in Neova Online in the browser. The Free plan crawls 1,000 URLs with every check; see Pricing or download Neova.

Questions

What is llms.txt?

A Markdown file at /llms.txt that names a site, summarizes it and lists the pages a language model should read. It was proposed on llmstxt.org on September 3, 2024, and is not an official web standard.

Does Google use llms.txt?

No. Google's guide to generative AI features, published May 15, 2026, says Google Search ignores llms.txt and that it neither helps nor harms visibility there.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may fetch and is a standard, RFC 9309. llms.txt only suggests what to read and blocks nothing.

How many sites have an llms.txt?

In SEO ProCheck's September 29, 2026 measurement, 31 of 184 top sites served /llms.txt (16.8%), nearly all of them software or developer tool companies.

What is llms-full.txt?

A companion file that holds the full text of the listed pages in one Markdown file instead of links to them. In the same measurement, 6 of 184 sites served one.

Take this with you

The assistant links open with a question about this page already written.

Crawl your site with NeovaThe Free plan runs every check on Mac, Windows and Linux. No credit card required.

Keep Reading