What Is llms.txt and Should Your Website Have One?

What Is llms.txt?

llms.txt is a markdown file you place at the root of your website, at /llms.txt, that gives large language models a short summary of your site and links to its most useful pages. Jeremy Howard of Answer.AI proposed it in September 2024 as a way to offer AI models clean, readable content instead of cluttered HTML.

It's a reading guide, not a rulebook. llms.txt contains no instructions or permissions, so it can't tell AI systems what to do with your content.

Should you have one? It's cheap to add, but no major AI search platform has committed to reading it. Whether it's worth your time depends on who visits your site.

How an llms.txt File Is Structured

An llms.txt file is ordinary markdown served as plain text at /llms.txt. You could write one in any text editor in a few minutes. It contains three things: the name of your site, a short summary of what it offers, and grouped lists of links to the pages that matter most, each with a one-line note. Because it's plain markdown, a person can skim it and a model can load it into its context without stripping out menus, scripts or ads.

The Required and Optional Parts

The llmstxt.org specification requires exactly one element: an H1 with the name of the site or project. Everything else is convention, meant to help a model decide what to read first:

• a blockquote with a short summary
• optional free-text notes
• H2 sections listing links, each with a brief description

# Example Co
> Invoicing software for freelancers.

## Docs
- [Quick start](https://example.com/start.md): setup in five steps
- [API reference](https://example.com/api.md): endpoints and auth

What Changed in Version 2

In August 2026 the spec moved to version 2, revised after two years of real-world use. The changes matter if you're publishing one now:

• Markdown page versions. A page can offer a clean markdown copy by appending .md to its URL or replacing the extension.
• Discovery tags. rel="alternate" type="text/markdown" points to that copy, and rel="describedby" points to the llms.txt file.
• No special “Optional” section. It's now just a naming convention.

llms.txt vs robots.txt vs sitemap.xml

Three files can sit at your domain root, and they do three different jobs. Only one of them tells crawlers what they may do, and only one is an official internet standard. Only llms.txt is written for a model to read rather than a crawler to follow.

You can check the first one with a free AI robots.txt checker.

What Each File Actually Does

robots.txt tells crawlers where they may go. Even that is a request: RFC 9309 states that its rules are not a form of access authorization.

sitemap.xml lists URLs so search engines can discover them. It aims for completeness, not judgement.

llms.txt does neither. It holds no directives and doesn't try to list everything. It's an editor's shortlist: here is what this site is, and here are the pages worth reading first.

How Their Adoption Curves Compare

• robots.txt dates to 1994 and became an IETF standard, RFC 9309, in September 2022.
• Sitemaps launched with Google in June 2005. By November 2006, Google, Yahoo! and Microsoft announced joint support.
• llms.txt was proposed in September 2024. Two years later, no major AI search platform has committed to reading it.

Publishing is growing fast anyway: Originality.ai counted 4,088 files in June 2025 and 36,120 by May 2026. The 10-site AI visibility study found most sites still lack one.

Which AI Platforms Actually Read llms.txt?

Publishing an llms.txt file and reading other sites' files are different things. Several AI companies publish one for their own developer documentation, including OpenAI at developers.openai.com/llms.txt. That tells you they find the format useful for their docs, not that their crawlers look for yours.

No major AI search platform has said it reads site owners' llms.txt files when answering questions. What drives AI citations is covered in how to get cited by ChatGPT; llms.txt isn't on that list.

What Google and the AI Companies Say

• Google Search says in its AI optimization guide that you don't need AI text files to appear in Search, including its AI features, because Search doesn't use them. Adding one neither helps nor harms.
• John Mueller of Google said in June 2025 that no AI system currently uses llms.txt.
• OpenAI's crawler documentation handles opt-outs through robots.txt and doesn't mention reading your llms.txt.
• Chrome Lighthouse now audits llms.txt, but marks a missing file Not Applicable because it's optional.

What Server Logs and Citation Studies Show

Three independent studies point the same way:

• Ahrefs looked at 137,210 domains: 28% publish a valid llms.txt, and 97% of those files got zero requests in May 2026. SEO audit tools were the biggest requester.
• SE Ranking analyzed about 300,000 domains and found no correlation between llms.txt and AI citations.
• Semrush saw no visits to Search Engine Land's file from GPTBot, ClaudeBot, PerplexityBot or Google-Extended over about ten weeks.

Where the file does get read, Ahrefs found, it's mostly by coding agents reading developer docs.

How to Create an llms.txt File

Creating the file takes less time than reading about it. On a small site, write it by hand. On a large site or a documentation platform, let your CMS generate it and then edit the result.

Either way, treat it as an editorial summary rather than an export of your sitemap. A model reading it should come away knowing what you do and which few pages answer the questions people ask about you. Keep it consistent with robots.txt, too: don't list pages you block.

Write the File

1. Start with an H1 containing your site or product name.
2. Add a one-sentence blockquote saying what the site is and who it serves.
3. Group your most useful pages under a few H2 headings, such as Docs, Guides or Pricing, with a one-line note after each link.
4. Link to canonical, indexable URLs only.
5. Keep it short. Pick the pages you'd want an AI to quote, not every page in your sitemap.

Publish It and Check It

1. Upload the file to your domain root so it loads at /llms.txt as plain text.
2. On WordPress, Yoast SEO can generate it and refresh it weekly. Documentation platforms such as Mintlify host one automatically.
3. Make sure robots.txt doesn't block the file.
4. Update it when key pages change.

AEO Tester checks for llms.txt alongside 25 other AI visibility signals: whether the file exists, whether robots.txt blocks it, and whether it follows the spec format.

llms.txt FAQ

These are the questions that come up most once teams decide to try llms.txt: whether they have to, whether it protects their content, and what the companion llms-full.txt file is for. Each answer stands on its own, so you can jump straight to the one you need.

Is llms.txt mandatory?

No. No search engine or AI platform requires it. Google says Search ignores AI text files, so adding one neither helps nor harms your rankings, and Lighthouse treats a missing file as Not Applicable rather than a failure. Nothing on your site breaks without it. Treat it as an optional extra for readers that choose to look for it, mainly coding agents today.

Does llms.txt stop AI companies from training on my content?

No. llms.txt contains no directives, so it can't block anything. If anything, it makes your key pages easier to find. Listing a page there doesn't grant or deny access; robots.txt and your server rules still decide who can fetch it. Training opt-outs live in robots.txt, using each company's crawler name, for example:

User-agent: GPTBot
Disallow: /

Check other AI companies' crawler documentation for their user-agent names.

What is llms-full.txt?

llms-full.txt is a companion file that bundles a site's full content, usually documentation, into one large markdown file, so an agent can load everything at once. It's a convention popularized by documentation platforms such as Mintlify and GitBook, not part of the llmstxt.org specification. Version 2 of the spec leans on per-page markdown copies instead. It's mostly worth it for docs sites.

Should Your Website Have an llms.txt File?

It depends on who reads your site:

• Documentation, APIs and developer products: yes. Coding agents are the one group the data shows actually requesting the file, so it has a real reader.
• Content, marketing and e-commerce sites: optional. It takes minutes and does no harm, but don't expect a measurable lift in AI visibility.

Either way, don't put it ahead of the work that does move AI visibility: open crawler access, clean structure and pages with clear answers. If a major AI platform commits to reading llms.txt, move it up your list. Until then, publish one if it's cheap for you, and put your effort into the pages themselves.