In short
So far, the evidence says that llms.txt has little effect. In Ahrefs' study of 137,210 domains, 97% of valid llms.txt files got no requests during the month, and AI retrieval bots made up only about 1% of requests. Google has reportedly said it isn't needed for its AI features. It costs little to add, and can help developer tools read documentation, but don't expect it to raise your AI visibility.
What it's meant to do
llms.txt is a proposed plain-text Markdown file at the root of your site, such as example.com/llms.txt. It's supposed to give AI systems a clean summary of your site, along with links to key pages. We've explained how to create one in how to build an llms.txt file.
The idea makes sense: web pages are full of menus and scripts, and a curated map would be easier for a model to process. But a standard only works if somebody uses it. No major AI company has confirmed that it relies on llms.txt as a ranking or citation signal for its search products. OpenAI's crawler documentation, for instance, covers robots.txt and its bots, and doesn't identify llms.txt as a signal.
What the data says
| Source | Sample | Finding |
|---|---|---|
| Ahrefs, via Search Engine Journal | 137,210 domains using Ahrefs Web Analytics | 97% of valid llms.txt files had no requests in the study month |
| Same study | Requests that did occur | About 1% from AI retrieval bots; SEO audit tools were the most common requesters, at 21.7% |
| Optimizely's review | Summary of several studies | Rarely requested; no shown effect on citations |
| Pixis's review | Citation data | Small differences, not treated as meaningful evidence of an effect |
Ahrefs' study has one important limit, which it acknowledges: it measured requests, and not whether a fetched file affected an answer. And crawling behaviour may change. But a file that hardly ever gets read isn't likely to be doing very much.
Some of the smaller studies you'll see, including those reporting low request counts, point in the same direction, although their methods vary and some come from vendors with an interest in the outcome.
What Google says
Google has been fairly clear on its own products. Its AI features guidance says that there are no additional requirements for appearing in AI Overviews or AI Mode. Secondary reports on its May 2026 guide say that it listed llms.txt among the things you don't need, and that it won't harm or help your visibility in Search.
That covers Google only. Other engines could behave differently, and that's where the log studies come in.
Where it can still be useful
Developer documentation. This is the strongest case. A coding assistant or an IDE plugin may be pointed at an llms.txt, or at a full-text version of your docs, and Markdown is easy for it to consume. If developers use your product through AI tools, then giving them a clean version of your docs makes sense. However, this tends to be a human or a tool deliberately fetching the file, and not a search crawler discovering it.
Agents acting on instruction. A person could tell an agent to read your llms.txt. That's a narrow use for now, though it may grow. See how to make your website agent-ready.
A useful exercise. Writing a short summary of what your site is, and which pages matter most, is good discipline, and it may improve your actual pages.
Very little downside. It's just a text file. The risk is mainly in spending too much time on it, or in paying for it.
Should you add one?
| You are | Recommendation |
|---|---|
| A software company with developer docs | Yes, it's cheap and useful |
| A site with a CMS that generates one automatically | Fine, leave it on |
| A small business hoping for AI visibility | Optional; do it last |
| Being sold it as a paid AI search service | No |
| Publisher wanting to control AI use | It won't do that; see robots.txt and licensing |
If you do add one, keep it accurate. A stale file with old prices or dead links is worse than no file. Don't use it to show AI systems different information from what your site says, either.
It also doesn't control access. Unlike robots.txt, it isn't meant to grant or deny anything. To manage AI crawlers, see our AI crawler user agents list.
What to do instead
If AI visibility is the goal, these have a much stronger case:
- Check access. Make sure that your firewall isn't blocking AI search crawlers.
- Serve content in your HTML, because not all bots run JavaScript. See why AI-built websites struggle in Google.
- Answer questions directly. See how to write content that AI answer engines quote.
- Add original facts, such as data, prices and first-hand tests.
- Earn third-party mentions. See how to get your brand cited by ChatGPT, Claude and Gemini.
- Measure the results. See how to measure whether AI engines are citing you.
Test it on your own site
You don't have to take anyone's word for it. If you already have the file, search your server or CDN logs for requests to /llms.txt, and note who's requesting it. Compare with requests for /robots.txt. After a month, you'll know whether anybody reads yours. We publish one ourselves, and this kind of check is how we'd judge its value.
If you'd like help working out what's really worth doing for AI search, book a call.
Common questions
Do AI companies use llms.txt?
No major AI search product has confirmed relying on it as a ranking or citation signal. Server-log studies show very few requests from AI retrieval bots. Some developer tools can use it for documentation, particularly when somebody points them at it deliberately.
Does llms.txt help with SEO?
There's no evidence that it does. Google has reportedly said that it won't help or harm your visibility in Search, and its guidance says no special files are needed for AI features. Focus on crawlable, useful content instead.
Is llms.txt worth adding?
If you have developer documentation, or if your CMS can generate it cheaply, it's fine to add. For most small businesses, it's a low priority. Don't pay anyone for it as an AI visibility service, because the evidence doesn't support that.
Can llms.txt block AI crawlers?
No. It's a summary file, not an access control. To manage AI crawlers, use robots.txt rules for each agent, and server or firewall rules if you need to enforce the block, because some user-triggered fetchers don't follow robots.txt.