Should You Block AI Crawlers?

Two different crawlers, two different decisions

The conversation usually treats this as one choice. It is not.

Training crawlers gather content to train or improve models. Your page becomes part of a dataset. There is no direct traffic and no attribution.

Retrieval crawlers fetch pages in real time to answer a specific question, and typically cite the sources they used. This is how a user asking an assistant about your industry ends up with a link to your article.

Major operators publish separate user-agent names for these purposes. Blocking the wrong one is the common error — you can remove yourself from AI answers while leaving training access untouched, which is precisely backwards for most businesses.

What blocking training costs you

Almost nothing directly. There is no traffic from training, no citation, no attribution.

The argument for allowing it is indirect and speculative: content in training data may make a model more likely to know about you. That effect is unmeasurable and unreliable, and it is not a business case.

So if you object to your work being used as training material — a reasonable position, particularly for original research or creative work — blocking training crawlers costs you very little.

What blocking retrieval costs you

This one is expensive, and growing more so.

When someone asks an assistant a question your content answers, retrieval determines whether you can be cited. Block it and you are absent from that answer. A competitor who allowed it is cited instead.

Given that a substantial and increasing share of discovery happens through AI-mediated answers, this is not a marginal channel any more. Blocking retrieval is effectively opting out of a search surface — the equivalent of blocking a search engine.

For any business that treats search as a route to customers, blocking retrieval is difficult to justify, and it is frequently done accidentally by teams who intended to block training.

How to decide

Allow retrieval if you want to be found. Nearly everyone selling something does.

Consider blocking training if your content is the product — original research, licensed material, creative work — or if you object on principle. It costs little.

Allow both if you want maximum reach and have no objection to training use. This is a defensible default for a marketing site.

Block both only if your content genuinely should not be publicly accessible, in which case it probably should be behind authentication rather than relying on a text file.

The practical position for most businesses: block training, allow retrieval, and check periodically that the agent names have not changed.

Implementing it

In robots.txt, per user agent. The mechanics are the same as for any crawler, and the same caveats apply — robots.txt controls crawling rather than indexing, and it is a request that well-behaved crawlers honour rather than an enforcement mechanism.

Two practical points:

Agent names change. Operators introduce new ones, rename, and split purposes. A configuration written a year ago may not cover current agents, so this needs occasional review rather than being set once.

Verify with your logs. Check which agents actually request your pages and how often. That tells you what is really happening rather than what you assume, and it will usually surface agents you had not considered.

If you want a stronger boundary than a voluntary instruction, that requires authentication or blocking at the network level — and blocking by IP or user agent at the edge is genuinely enforceable in a way robots.txt is not.

The strategic point

The framing that matters: for most businesses, AI answer engines are a distribution channel, not a threat to be defended against.

Content that gets cited brings qualified visitors who arrived because a system judged you the best answer. That is a good position, and it is available to sites that allow retrieval and structure their content well.

The teams doing well here are not the ones blocking. They are the ones making their material easy to cite — direct answers, sourced figures, clear structure, original data — and then allowing the crawlers that would cite it. That is the same work as being cited by AI, and blocking retrieval throws it away.

If you are unsure what is currently crawling your site and what you are allowing, book a call — the log check takes an hour and the answer is often not what people expect.

Common questions