robots.txt

robots.txt controls crawling, not indexing. A URL blocked in robots.txt can still appear in Google's results, usually without a description, if other pages link to it. To keep a page out of search results you need a noindex tag, and the page must not be blocked, or the crawler never sees that tag.

It is also not security. The file is public, and well-behaved bots follow it voluntarily. Anything private needs a password or must not be on the public web at all.

How it works

The file lives at one address, such as example.com/robots.txt, and applies only to that host and protocol, so a subdomain needs its own. It contains groups of rules: a User-agent line naming a crawler (or * for all), then Disallow and Allow lines listing paths. It can also include the address of your XML sitemap.

Before crawling a site, a compliant bot fetches robots.txt and skips disallowed paths. Rules can target particular bots, which is how site owners allow search crawlers but block specific AI crawlers, though only bots that respect the file will obey. Each crawler reads the group of rules that names it most specifically and ignores the others, so a bot named in its own group does not also follow the rules written for all bots. That catches people out when they add a rule for one crawler and accidentally lift the general restrictions for it. Changes are not instant either: Google generally caches the file for up to a day.

Rules match from the start of the URL path, so "Disallow: /admin" blocks /admin, /admin/settings and also /administration. Google follows the most specific matching rule, and where an Allow and a Disallow rule are equally specific it applies the less restrictive one. Google supports * as a wildcard and $ to mark the end of a URL. A missing file (a 404) is read as no restrictions, but if the server returns an error such as a 5xx when Google asks for the file, Google pauses crawling the site and may fall back on its last copy until it can read the file again.

Example

A clinic's booking system creates thousands of calendar URLs with date parameters, and Googlebot spends time crawling empty slots. A Disallow rule for the calendar path stops that, while the main service pages stay open. Separately, the staff login page gets real password protection, because listing it in robots.txt would only advertise where it is.

To check the setup, the clinic opens its robots.txt in a browser to confirm the new rule is live, then views the robots.txt report in Search Console to see that Google has fetched the latest version without errors. URL Inspection on a calendar URL now reports crawling as blocked, while a service page shows as allowed. A week later, Crawl stats show fewer requests to calendar URLs.

Why it matters for your business

A correct robots.txt keeps crawlers away from useless areas and lets you decide which bots you allow. A wrong one can be catastrophic: a single "Disallow: /" left over from a development site blocks crawling of the entire site. For most small sites the safest file is short, and the main job is making sure it does not block anything important. It is worth looking at the file after every launch or redesign, because that is when the most damaging mistakes appear.

Common mistakes

• Leaving a development-site rule that blocks the whole site after launch.
• Using robots.txt to hide pages from search results instead of a noindex tag.
• Blocking CSS and JavaScript files that search engines need to render the page.
• Treating robots.txt as a way to protect private or sensitive pages.

Questions people ask

Does robots.txt stop a page from being indexed?

No. It stops compliant crawlers from fetching the page, but Google can still index the URL if it finds links to it, typically showing it without a description. To keep a page out of results, allow crawling and add a noindex tag, or protect the page with a password.

Where do I find my robots.txt file?

Add /robots.txt to your domain, for example example.com/robots.txt. If nothing loads, the site has none, and crawlers will assume everything may be crawled, which is fine for many small sites. Google Search Console has a robots.txt report showing the version Google last fetched and any errors.

Can robots.txt block AI crawlers?

It can ask them not to crawl. Many AI companies publish crawler names that respect robots.txt rules, so you can disallow those user agents specifically. Compliance is voluntary, though, and blocking a crawler affects only future crawling, not content already collected. Decide based on whether you want to appear in AI answers.