In short
robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it, because the crawler is told not to fetch it but not told to exclude it. To keep a page out of the index, allow crawling and use a noindex directive. Never use robots.txt to hide sensitive content — the file is public and lists exactly what you wanted hidden.
What it is
A plain text file at the root of your domain telling crawlers which paths they may fetch. It must live at /robots.txt — nowhere else is read.
It is a set of instructions that well-behaved crawlers follow voluntarily. It is not access control, and crawlers that ignore it face no obstacle.
Crawling is not indexing
This is the distinction almost every robots.txt mistake comes from.
Disallow prevents fetching. It does not prevent indexing.
If you disallow a page and other sites link to it, search engines may still index it — showing the URL, often with no description, because they were not permitted to read it. The page appears in results looking broken, which is worse than either alternative.
So:
To keep a page out of search, allow crawling and use a noindex meta tag or header. The crawler must fetch the page to see the directive, which is why blocking it is counterproductive.
To save crawl budget, disallow. That genuinely prevents the fetch, which is the point on large sites.
Doing both to the same URL is the common error: the crawler cannot fetch the page, so it never sees the noindex, so the URL may remain indexed indefinitely.
Basic syntax
Groups begin with User-agent, followed by rules. Disallow blocks a path prefix; Allow creates an exception within a blocked path.
An empty Disallow: means nothing is blocked. Disallow: / blocks the entire site — that single character is the line that removes sites from search, and it usually arrives when a staging configuration is deployed to production.
Paths are prefixes, so Disallow: /admin blocks /admin, /admin/, and also /administrator. Be precise about trailing slashes.
Most crawlers support * as a wildcard and $ to anchor the end, which is how you block file types or parameter patterns.
Add a Sitemap: line with the absolute URL. It is the simplest way to make your sitemap discoverable.
What it is genuinely useful for
Blocking infinite or generated spaces — calendars, filter combinations, internal search results — so crawlers do not spend effort there.
Blocking parameter noise that produces duplicate content.
Keeping crawlers out of heavy endpoints that cost you money or time to serve.
Pointing at the sitemap.
Never use it to hide things
robots.txt is public. Anyone can read yours, and it is one of the first things anyone assessing a site looks at.
Disallowing /admin-secret-panel/ publishes the existence and location of your admin panel to everyone, while providing no protection whatsoever — it is an instruction, not a barrier.
Use authentication for anything sensitive. For staging environments, use HTTP authentication rather than a disallow, since a disallowed staging site can still end up indexed via a link.
AI crawlers
Most major AI companies now publish user-agent names and respect robots.txt, so you can allow or block them selectively.
The decision is genuinely yours and depends on what you want. Blocking them protects content from being used in training. Allowing them keeps you eligible to be surfaced and cited in AI answers, which is an increasing share of how people find things.
Be aware the two are often separate: some operators distinguish between crawling for training and crawling for real-time answer retrieval, with different agent names for each. Blocking the wrong one can remove you from AI search results while doing nothing about training.
If visibility matters more than exclusivity — which for most businesses it does — allowing retrieval while considering training separately is the more common position.
The mistakes that cause real damage
Disallow: / in production. Almost always a staging file deployed by accident. Check this first if traffic collapses after a release.
Blocking CSS or JavaScript. Search engines render pages, and blocking the assets means they see something broken.
Blocking a page you also noindexed, so the directive is never read.
Assuming it removes existing indexed pages. It does not — it prevents future crawling, which can leave stale results in place. Use noindex, or removal tools, for content already indexed.
Case sensitivity. Paths are case-sensitive; /Admin and /admin are different.
Blocking your own sitemap.
Checking it
Fetch yourdomain.com/robots.txt and read it. It is surprising how often nobody has.
Then use Search Console's robots.txt report to see the fetched version and whether specific URLs are blocked. And after any deploy that touches infrastructure, check it again — this file changes by accident more often than by intention.
If your traffic dropped suddenly after a release, this file is the first place to look — book a call.
Common questions
Does robots.txt stop a page appearing in Google?
No. It controls crawling, not indexing — a disallowed URL can still be indexed if other pages link to it, appearing in results with no description because the crawler was not allowed to read it. To keep a page out of search, allow crawling and use a noindex directive instead.
What is the difference between robots.txt and noindex?
Disallow in robots.txt prevents the page being fetched, which saves crawl budget but does not guarantee exclusion from results. Noindex prevents indexing but requires the page to be fetched so the directive can be read. Applying both to one URL means the noindex is never seen.
Can I use robots.txt to hide private pages?
No — the file is public and one of the first things anyone assessing a site reads, so disallowing an admin path publishes its location while providing no protection. Use authentication for anything sensitive, and HTTP authentication rather than a disallow for staging environments.
Should I block AI crawlers in robots.txt?
It depends on whether you value exclusivity or visibility. Blocking protects content from training use; allowing keeps you eligible to be cited in AI answers. Note that some operators use different agent names for training and for real-time retrieval, so blocking the wrong one can remove you from AI results without affecting training.
What is the most damaging robots.txt mistake?
Disallow: / in production, which blocks the entire site and usually arrives when a staging configuration is deployed by accident. If organic traffic collapses after a release, check this file first. Blocking CSS or JavaScript is also damaging, since search engines render pages and will see something broken.
