In short
An llms.txt file is a plain-text index at your site's root that lists your most important pages in a short, structured format so AI crawlers can find your best content without guessing. Keep it short, keep it current, and link only to pages worth citing.
What llms.txt actually is
llms.txt is a plain-text file at your domain root (/llms.txt) that gives AI crawlers a short, curated map of your site: a one-line description of who you are, followed by grouped links to your most important pages. It's the same idea as robots.txt or sitemap.xml, but written for a language model instead of a search indexer or a crawler's rule engine — prose-like, not markup.
Why it's worth doing
A sitemap tells a crawler everything exists. llms.txt tells a model what matters. Given a limited budget of pages to fetch, a curated shortlist increases the odds your best material — not a stale tag page or an old landing page — is what gets read and cited. It's a small file with an outsized signal-to-noise ratio.
The structure that works
Keep it to four parts:
- A one-line identity statement. Who you are and what you do, in plain English.
- A short paragraph of context. What kind of company, what you're known for, one or two sentences.
- Grouped links under
##headings. Group by topic or content type — Docs, Guides, Case studies — not by date. - A one-line description per link. Not the page title restated; what a reader gets from clicking it.
Skip anything meant for humans only — no marketing copy, no calls to action. This file is read by a model deciding whether to fetch a page, not by a visitor deciding whether to buy.
What to include, and what to leave out
Include your flagship guides, your core service/product pages, and anything you'd want quoted if someone asked an AI assistant about your category. Leave out: login pages, thank-you pages, paginated archives, and anything thin or duplicate. If a page isn't good enough to hand a journalist, it isn't good enough for llms.txt.
Keep it current
A stale llms.txt is worse than none — it points a crawler at content that no longer represents your best work. Update it whenever you publish a genuinely new flagship piece, and prune it whenever an older page is superseded. Treat it like a curated "best of," not an automatically generated dump — if you're publishing more than a couple of times a week, wire the update into your publishing step so it never drifts out of date. (See how we drip-publish and index new articles for the mechanics.)
A minimal example
``` # Troiana
> Design and development studio. We build product design, brand identity, and engineering for companies that care how things are made.
Guides - The Complete Guide to GEO: what generative engine optimization is and how to do it. - What Makes a Website Fast: the performance fundamentals that actually move the needle.
Services - Generative Engine Optimization: getting cited by AI answer engines. ```
Common mistakes
The most common mistake is treating llms.txt like a sitemap and dumping every URL into it — that defeats the purpose. The second most common is writing it once and never touching it again. The third is filling the descriptions with keyword-stuffed marketing language instead of a plain, accurate sentence — models weight clarity over persuasion.
Common questions
Is llms.txt an official standard?
It's an emerging, informally-adopted convention rather than a ratified spec — support varies by crawler. It costs almost nothing to add and has clear upside, which is why it's worth doing even before it's universally supported.
Does llms.txt replace sitemap.xml?
No. Keep both. sitemap.xml is a complete, mechanical index for search crawlers; llms.txt is a short, curated shortlist for AI crawlers deciding what's worth reading.
How long should llms.txt be?
Short. A few dozen links at most, grouped under clear headings. If it's long enough to need a table of contents, it's too long.