llms.txt
llms.txt is a plain-text file at the root of a website (https://yourdomain.com/llms.txt) that lists the key pages and content an AI language model should read to understand the site.
llms.txt is a plain-text file at the root of a website (https://yourdomain.com/llms.txt) that lists the key pages and content an AI language model should read to understand the site. Where robots.txt tells crawlers which paths to avoid, llms.txt tells language models which paths are worth reading.
The format was proposed in late 2024 as a way to help AI models extract a site's most important content efficiently without crawling thousands of low-value pages. It uses Markdown — H1 for the site name, H2 for sections (e.g. "Core pages", "Docs"), and Markdown links for each entry. A typical llms.txt file is 30–100 lines.
Adoption is uneven but growing. ChatGPT, Claude, and Perplexity have all been observed fetching llms.txt during page-discovery passes. Google has not formally endorsed the format; Gemini's crawler does not request it.
The benefit for site owners is two-fold. First, AI models that respect llms.txt are more likely to surface the pages you actually want them to cite (rather than indexed-but-stale archive pages). Second, the discipline of writing the file forces an honest review of which pages are worth a language model's attention — most sites discover that 80% of their indexed URLs are not citation-worthy.
AEOlens checks llms.txt presence and structural quality as part of its 48-check structural audit. A free llms.txt generator at aeolens.ai/tools/llms-txt-generator produces a starter file from any domain.
Frequently asked
Where should the llms.txt file be located?
At the root of your domain: https://yourdomain.com/llms.txt. Optionally also at /.well-known/llms.txt. Both locations are recognised by current implementations.
Which AI engines respect llms.txt?
As of 2026, ChatGPT, Claude, and Perplexity have been observed fetching llms.txt during crawls. Google has not endorsed the format. Adoption is growing; the format costs nothing to add, so the recommended posture is to ship it.
How is llms.txt different from sitemap.xml?
Sitemap.xml lists every URL on the site for traditional search crawlers. llms.txt curates the small subset (typically 10–50 URLs) most worth a language model's attention — usually pricing, docs, key product pages, and core blog content. They serve different audiences.
Does llms.txt replace robots.txt?
No. robots.txt tells crawlers what they may not fetch (with Disallow rules). llms.txt suggests what AI models should fetch (with curated links). They are complementary.
See how AI engines read your own site
Run a free AEO audit on any URL — 48 structural checks, score in under 60 seconds, no signup.
Run a free audit