In short
Bots visit websites to index them for search, to gather content for AI training and answers, to monitor uptime and prices, to check links and SEO, to scrape content and email addresses, and to probe for vulnerabilities. Industry measurements put automated traffic at roughly half of all web requests. Welcome the search and AI-search crawlers you want to be found by, tolerate monitors and legitimate tools, and block or rate-limit scrapers and scanners with a firewall or CDN rules. Robots.txt is a request that good bots honour and bad bots ignore.
How much of your traffic is bots
Security and CDN vendors who see a large share of web traffic have reported for years that automated requests make up somewhere between 40% and 50% of the total, with a growing portion classified as malicious. On a small site the share can be higher, because human traffic is low and the bots visit regardless. If your server logs show far more requests than your analytics shows sessions, the difference is mostly bots, since most of them do not run the JavaScript that analytics depends on.
The kinds of bot, and what they want
Search engine crawlers. Googlebot, Bingbot, and their smaller peers fetch pages to index them. This is the traffic that makes your site findable. They identify themselves honestly, obey robots.txt, and can be verified by reverse DNS lookup.
AI crawlers. Two distinct purposes: training data collection (GPTBot, ClaudeBot, Google-Extended, CCBot) and real-time retrieval for answers (OAI-SearchBot, PerplexityBot, and the search-mode agents). The second kind is how a site gets cited by AI assistants; the first is a policy choice. Whether to block AI crawlers is a separate decision from blocking bots in general.
Monitoring and tools. Uptime checkers, performance testers, SEO tools crawling for audits, link checkers, RSS readers, social platforms fetching link previews, and archive services. Mostly harmless and often useful; some SEO tools crawl aggressively enough to be worth rate-limiting.
Scrapers. Software copying your content to republish it, harvesting prices, collecting email addresses, or building datasets. Ranges from competitors' price monitors to spam operations. They usually lie about who they are.
Scanners and attackers. Automated probes for known vulnerabilities: requests for /wp-login.php, /.env, /phpmyadmin, old plugin paths, and hundreds of others, on every site on the internet, constantly. Followed, when something is found, by exploitation attempts. Also credential-stuffing bots trying leaked passwords on any login form, and spam bots filling forms.
Click and ad fraud bots, which mostly matter for sites running ads.
Telling them apart
User agent. Every request announces a user agent string. Legitimate bots name themselves; a quick search identifies most. Scrapers and attackers commonly claim to be a browser, or a well-known bot.
Verification. Google, Bing and the major AI companies publish their IP ranges or support reverse DNS verification. A request claiming to be Googlebot from an address that does not resolve to Google is not Googlebot.
Behaviour. Bots request pages faster than a person could read them, fetch many pages with no images or scripts, hit paths that do not exist on your site, and arrive without the referrers and session patterns humans produce. Server logs show all of this; analytics shows almost none of it.
Analytics anomalies. Sessions of zero seconds, visits from data-centre locations, spikes with 100% bounce rate, and referrer spam are bot traffic that got through to analytics.
What to allow and what to block
Allow search crawlers and the AI retrieval crawlers you want to be cited by. Blocking these makes the site invisible in the channels they serve.
Allow with limits monitoring services, social preview fetchers, and legitimate SEO tools. If one is hammering the site, robots.txt Crawl-delay (honoured by some) or a rate limit at the CDN.
Block or rate-limit scrapers and scanners. A CDN or web application firewall such as Cloudflare's does most of this automatically: known bad user agents, data-centre ranges with no legitimate purpose, request rates no human produces, and requests for paths that only attackers ask for. Turning on a managed bot rule set is usually enough for a small site.
Do not try to block everything that is not a browser; you will block the crawlers you need and the fetchers your own social posts rely on.
Robots.txt and its limits
Robots.txt tells well-behaved bots which paths to avoid. Search engines, AI companies and reputable tools honour it. Scrapers, scanners and attackers ignore it, and it is public, so listing /admin/ in it is an invitation. Use it for crawl guidance; use a firewall, authentication and rate limiting for anything that must actually be enforced.
Keeping bots out of your numbers
Enable the bot filtering option in analytics, which removes known crawlers. Filter referrer spam by hostname. Watch for data-centre traffic and zero-second sessions in reports and exclude the sources. Accept that some will always leak through, and that some human traffic with tracking blocked will always be missing, so analytics is a sample rather than a census.
The practical setup
A CDN with bot management in front of the site; robots.txt allowing the crawlers you want; rate limiting on forms and login endpoints; monitoring that alerts on unusual request volumes; and periodic reading of the server logs, where the bots are honest about what they are doing even when they lie about who they are. If your logs are showing traffic you cannot account for, book a call and we will classify it.
Common questions
Are bots bad for my website?
Some are essential and some are hostile. Search and AI-retrieval crawlers are how your site gets found and cited. Monitors and preview fetchers are useful. Scrapers copy your content, scanners probe for vulnerabilities, and spam and credential-stuffing bots attack forms and logins. The job is to welcome the first groups and block or rate-limit the last.
How do I know if traffic is from a bot?
Check the user agent string in server logs, verify claimed crawlers such as Googlebot by reverse DNS or published IP ranges, and look at behaviour: page requests faster than reading speed, no images or scripts fetched, requests for paths that do not exist, and sessions of zero seconds in analytics. Most bots never appear in analytics at all because they do not run JavaScript.
Should I block bots with robots.txt?
Use robots.txt to guide well-behaved crawlers away from paths they should not index, but not to enforce anything. It is a public request that search engines and reputable tools honour and that scrapers and attackers ignore. Enforce with a CDN firewall, rate limiting and authentication.
Why do bots request wp-login.php on a site that is not WordPress?
Because scanners probe every site on the internet for common vulnerabilities regardless of what it runs, requesting hundreds of known paths in the hope one exists. Requests for wp-login.php, .env, phpmyadmin and similar on a non-WordPress site are automated reconnaissance. They are harmless if the paths do not exist and are worth blocking at the firewall to reduce noise.
How do I stop bots from skewing Google Analytics?
Enable the known-bot filtering setting, exclude referrer spam by hostname, watch for spikes of zero-second sessions or data-centre traffic and filter the sources, and put a CDN with bot management in front of the site so much of it never reaches the page. Some leakage is inevitable; treat analytics as a sample.
