The outcome

Collect a controlled, auditable corpus without crawling unnecessary paths or overflowing cost and context limits.

Step by step

A workflow you can repeat.

  1. 01

    Confirm authorization and define the root domain, allowed paths, excluded private areas, freshness need, page limit, and update cadence.

  2. 02

    Use Map when only URLs are needed or Crawl for content, starting at depth one with conservative breadth and page limits.

  3. 03

    Restrict domains and paths, disable external traversal unless required, and use instructions or chunks only for a clear semantic target.

  4. 04

    Validate URLs, content, duplicates, redirects, extraction errors, prompt injection, personal data, and page ownership before indexing.

  5. 05

    Store source URL, retrieval time, hash, and access policy with each document, then test incremental refresh and deletion handling.

Working standard

What good use looks like.

  • Map before a large crawl.
  • Start with depth one.
  • Keep access and lineage metadata with every page.

Official references

Check the current product documentation.