AI crawler

AI crawlers work like search crawlers but serve different ends. Some gather training data, some build an index for an AI search feature, and some fetch a page live because a user asked an assistant about it. Several companies use a separate user agent for each job, so blocking one does not necessarily block the others.

Examples include GPTBot from OpenAI, ClaudeBot from Anthropic and PerplexityBot from Perplexity. Google-Extended is different: it is a robots.txt token rather than a separate crawler, and it controls whether content Google collects may be used to train its Gemini models and to ground answers in its Gemini apps, without affecting Google Search. The list of agents and what each does changes, so check each company's current documentation.

How it works

Each crawler identifies itself with a user agent name in its requests, and the automatic crawlers of reputable companies follow the rules in robots.txt. Agents that fetch a page because a user asked are treated differently: OpenAI and Perplexity say robots.txt may not apply to those fetches, while Anthropic says its user agent follows it. A site owner can allow or disallow each agent by name, for the whole site or for specific folders. Some companies also publish the IP ranges their crawlers use, so a server can check that a visitor claiming to be a crawler is genuine.

Robots.txt is a voluntary standard. Well-known AI companies say they respect it, but it cannot stop a crawler that ignores it; that needs blocking at the server or firewall. Files such as llms.txt are proposals for guiding AI tools, not something crawlers are required to read.

To see which AI agents visit your site, look at your server's access logs, or your hosting or CDN dashboard, filtered by user agent. Because anyone can fake a user agent name, a request claiming to be GPTBot is only trustworthy if it comes from the IP ranges the company publishes. Some hosting and CDN providers offer settings to block or allow AI crawlers in bulk, which is convenient but can block more than you intend, so check exactly which agents each setting covers.

Example

A small publisher wants its articles to be cited in AI answers but does not want them used to train models. It reads each company's documentation and finds that some separate their training crawler from their search and user-triggered agents. It disallows the training agents in robots.txt, keeps the search agents allowed and checks its server logs a month later to confirm which agents are visiting.

The logs also show a crawler using a well-known agent name from an address outside that company's published ranges, requesting hundreds of pages a minute. The publisher blocks that traffic at the firewall rather than in robots.txt, since a fake crawler has no reason to follow the rules. The genuine agents keep visiting at their normal rate.

Why it matters for your business

Blocking AI crawlers protects content from some uses but can also keep a business out of AI answers, where some customers research their options, which affects AI visibility. Allowing them costs some server load and some control over reuse. Most small business sites gain more from being findable than they lose, while publishers whose content is the product face a harder trade-off.

Common mistakes

• Blocking every AI-related agent by habit and then wondering why AI assistants never mention the business.
• Assuming one robots.txt rule covers all of a company's agents, when training and search agents may be separate.
• Believing robots.txt stops crawlers that choose to ignore it.
• Copying a list of agent names from an old article instead of checking current documentation.

Questions people ask

Should I block AI crawlers on my website?

It depends on what you want. Blocking training crawlers limits how your content is used to build future models. Blocking the agents that power AI search and live fetching can keep you out of AI answers altogether. Many businesses allow the search agents and decide separately about training.

How do I block GPTBot?

Add a rule to your robots.txt file naming the agent: a line reading User-agent: GPTBot followed by Disallow: / blocks the whole site. OpenAI says GPTBot respects these rules. OpenAI's search agent has its own name and needs a separate rule, and OpenAI says robots.txt may not apply to pages fetched because a user asked.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended controls whether your content may be used to train Gemini models and to ground answers in Gemini apps, and Google says it does not affect Google Search, which includes AI Overviews. To limit how your pages appear there you would use search controls such as nosnippet, which also affect regular results.