llms.txt
llms.txt is a plain-text file placed at the root of a website that points AI crawlers and assistants to the pages and information you most want them to use.
It works like robots.txt or a sitemap, but for large language models: a simple, curated map of your most important content. It does not guarantee a citation, but it reduces the chance a model overlooks or misreads your best pages.
How it works
The file lives at /llms.txt and is written in Markdown: a title, a one-paragraph summary of the site, then sections of links, each with a short description of what the page covers. An assistant or crawler can read it in one request and understand what the site is about and where the authoritative pages are, without crawling everything.
It is a proposed convention rather than an official standard, so support varies by engine, but it costs little to publish and keep current.
Example
troiana.net's llms.txt is generated from the filesystem on every request: capabilities, insights, tools, glossary terms, the AI Hub and company pages are listed automatically, so a newly published article appears in it without anyone editing the file.
Common mistakes
- Listing every URL on the site instead of the pages that matter.
- Letting the file go stale so it points to removed or redirected pages.
- Expecting it to override robots.txt; it guides crawlers, it does not grant access.
