The outcome

Create a bounded website corpus whose source, lifecycle, and failures remain traceable.

Firecrawl is a web scraping, crawling, mapping, search, and structured extraction platform. Turning websites into markdown, structured records, URL maps, search results, screenshots, and asynchronous crawl datasets for AI applications. This guide narrows that broad capability into one repeatable outcome, with checkpoints that keep the source material and your judgment in the loop.

Before you begin

Set the boundary before the tool starts.

Choose one real task, identify who will use the result, and decide what evidence or test will make the result acceptable. Gather only the source material needed for that task. If the work contains confidential, personal, regulated, or client-owned information, confirm that the platform and account are approved before sharing it.

Troiana principle

AI should make the work easier to inspect. If the workflow removes the source, the owner, or the review step, redesign the workflow.

Step by step

A workflow you can repeat.

  1. 01

    Define allowed domains, paths, query handling, sitemap use, page and depth limits, data policy, refresh cadence, and stop conditions.

  2. 02

    Run Map first to inspect scope, then start a narrowly filtered crawl and persist its job ID and configuration.

  3. 03

    Configure an HMAC secret, verify each raw webhook signature, deduplicate by webhook ID, and handle retries and out-of-order events.

  4. 04

    Validate each page for canonical URL, status, duplicates, extraction errors, unsafe instructions, sensitive data, and indexing eligibility.

  5. 05

    Reconcile the completed job with retrieved results, record omissions and errors, then test update, deletion, cancellation, and cost controls.

Working standard

What good use looks like.

  • Inspect a map before crawling.
  • Verify webhook signatures.
  • Reconcile completion against stored pages.

Scraping can expose copyrighted, private, malicious, or personal data and browser actions can carry powerful credentials. Confirm authorization, protect keys and cookies, constrain scope, verify webhooks, manage caching and retention, and treat every retrieved page as untrusted input.

Official references

Check the current product documentation.

Features, plan limits, availability, and data controls change. These official pages are the starting points used for this collection.