The outcome

Create a bounded website corpus whose source, lifecycle, and failures remain traceable.

Step by step

A workflow you can repeat.

  1. 01

    Define allowed domains, paths, query handling, sitemap use, page and depth limits, data policy, refresh cadence, and stop conditions.

  2. 02

    Run Map first to inspect scope, then start a narrowly filtered crawl and persist its job ID and configuration.

  3. 03

    Configure an HMAC secret, verify each raw webhook signature, deduplicate by webhook ID, and handle retries and out-of-order events.

  4. 04

    Validate each page for canonical URL, status, duplicates, extraction errors, unsafe instructions, sensitive data, and indexing eligibility.

  5. 05

    Reconcile the completed job with retrieved results, record omissions and errors, then test update, deletion, cancellation, and cost controls.

Working standard

What good use looks like.

  • Inspect a map before crawling.
  • Verify webhook signatures.
  • Reconcile completion against stored pages.

Official references

Check the current product documentation.