Crawling

Crawling is only the first step. A crawled page has been fetched; it has not necessarily been understood, stored or ranked. Deciding whether to keep a page in the search engine's database is indexing, a separate step that can fail even after a successful crawl.

Crawlers are not limited to search engines. SEO tools, monitoring services and AI companies run their own bots, each with a name in its user agent that site owners can identify and, if they choose, restrict.

How it works

A crawler such as Googlebot starts from known URLs, including those in sitemaps and links found on pages already crawled. It checks robots.txt, requests the page, reads the response code and stores the HTML. For pages built with JavaScript, Google then renders the page in a headless browser to see the final content, which can happen some time after the first fetch.

New links found on the page join the queue. How often a site is recrawled depends on how often it changes, how important it seems and how quickly its server responds. Rendering is where many JavaScript-heavy sites run into trouble. If the important text and links only appear after scripts run, and those scripts fail or are blocked, the crawler may see a nearly empty page. Content in the initial HTML is the safest route to being crawled and understood, which is one reason server-side rendering and static pages tend to be easier for search engines.

Crawlers identify themselves with a user agent name, but names can be faked. Google publishes ways to verify that a visitor claiming to be Googlebot really comes from Google, such as a reverse DNS lookup or its published IP ranges. Crawlers also obey server signals: a 5xx server error or 429 Too Many Requests response makes Googlebot slow down. A site that is often slow or down is crawled less, which delays updates appearing in search.

Example

A new bakery website launches with no links from anywhere. Nothing appears in Google for a week. The owner submits a sitemap in Search Console and the local business association links to the site. Search Console's crawl stats soon show Googlebot visiting, and the URL Inspection tool reports the pages as crawled.

To check further, the owner opens Crawl stats in Search Console's settings. It shows total crawl requests per day, average response time, and a breakdown by response code and file type. A spike in server errors or very slow responses points to a hosting problem. URL Inspection's live test fetches a page as Googlebot would right now, and a screenshot shows whether the content renders correctly.

Why it matters for your business

If pages are not crawled, nothing else in SEO matters for them. Common blockers are robots.txt rules left from development, slow or failing servers, content only reachable through forms or scripts, and pages no other page links to. For most small sites crawling just works, and checking it is a quick health check rather than an ongoing task. For a site that changes often, slow crawling also means updated prices or opening hours take longer to reach search results.

Common mistakes

• Assuming a page is in Google simply because it has been crawled.
• Blocking crawlers with a leftover development rule in robots.txt.
• Hiding important content behind clicks, forms or scripts that crawlers cannot follow.
• Leaving important pages without any internal links, so crawlers rarely find them.

Questions people ask

How do I know if Google has crawled my page?

Use the URL Inspection tool in Google Search Console. It shows whether Google knows the URL, when it was last crawled, whether crawling was allowed and whether the page is indexed. The Crawl stats report gives a site-wide view of how often Googlebot visits and any errors it meets.

What is the difference between crawling and indexing?

Crawling is fetching a page; indexing is analysing it and storing it so it can appear in results. A page must be crawled before it can be indexed, but crawling does not guarantee indexing. Google may crawl a page and decide it is duplicate, low quality or blocked by a noindex tag.

How can I get Google to crawl my site faster?

Make sure pages are linked from your navigation and from each other, submit an XML sitemap, keep your server fast and reliable, and earn links from other sites. In Search Console you can request indexing of individual URLs, which prompts a crawl but does not guarantee the page will be indexed.