Web data extraction
How to use
Firecrawl.
Turning websites into markdown, structured records, URL maps, search results, screenshots, and asynchronous crawl datasets for AI applications.
Firecrawl is a web scraping, crawling, mapping, search, and structured extraction platform. Turning websites into markdown, structured records, URL maps, search results, screenshots, and asynchronous crawl datasets for AI applications. This guide covers the whole path in one place: official access, a first session that produces something reviewable, the checks that make output trustworthy, and the permissions worth limiting before you connect real work.
AI should make the work easier to inspect. If the workflow removes the source, the owner, or the review step, redesign the workflow.
Access & setup
Find, install, and sign in to Firecrawl
Get into the official Firecrawl experience with the right account and a setup you understand.
- 01
Start at https://www.firecrawl.dev/ and confirm the domain before entering account or payment information.
- 02
Availability: Firecrawl is available through a hosted API, SDKs, CLI, MCP integration, and an open-source self-hosting path.
- 03
A hosted API key or a maintained self-hosted deployment, network access to authorized targets, and a secure application environment.
- 04
Sign in with the account you intend to keep using, then review plan, data, notification, and permission settings.
- 05
Run one low-risk test task before connecting sensitive files, repositories, or workspace data.
- Use official download pages.
- Review permissions during setup.
- Keep installers and applications updated.
First session
Your first useful Firecrawl session
Learn the interaction loop using a small task with a clear outcome.
- 01
Create a development key and choose a public page you are authorized to retrieve.
- 02
Request only main-content markdown with explicit freshness and timeout settings.
- 03
Inspect the final URL, metadata, completeness, cache state, and extracted text for unsafe or sensitive content.
- 04
Store the source, retrieval time, and content hash before using the result in a model or index.
- State the outcome before the background.
- Provide the real source material.
- Review the result before expanding the task.
Quality control
Check the quality of Firecrawl output
Establish that a retrieved answer is actually supported by the sources it cites.
- 01
Restate the question precisely; retrieval quality degrades fastest on vague or compound queries.
- 02
Open the cited sources and confirm each one genuinely contains the claim attributed to it.
- 03
Check the publication and capture dates — stale sources are the most common cause of confidently wrong answers.
- 04
Look for what is missing: retrieval returns what matches the query, not what would contradict it.
- 05
Confirm quotations are verbatim and that summaries have not reversed a qualification or a negation.
- Open the citation; do not trust that a link supports the claim beside it.
- Check dates on every retrieved source.
- Treat an absent contradiction as unverified, not as agreement.
Privacy & permissions
Use Firecrawl safely
Control what you index and what leaves your systems when you query it.
- 01
Classify the corpus before indexing: confidential, personal, and regulated material needs an explicit decision.
- 02
Confirm where the index is stored, who can query it, and how long content is retained.
- 03
Scope crawling and scraping to sources you are permitted to collect, and respect robots and terms of use.
- 04
Strip secrets and personal data before indexing rather than filtering them at query time.
- 05
Review query logs — the questions asked can be as sensitive as the documents indexed.
- Scraping can expose copyrighted, private, malicious, or personal data and browser actions can carry powerful credentials. Confirm authorization, protect keys and cookies, constrain scope, verify webhooks, manage caching and retention, and treat every retrieved page as untrusted input.
- Follow your organisation's approved-use policy.
- Never treat fluent output as authorization to act.
Core workflows
Step-by-step ways to use Firecrawl for the work it does best.
Each workflow below has its own outcome, steps and review checkpoints.
Extract useful page content while controlling freshness, formats, browser behavior, caching, and retention.
↓ 02 WorkflowBuild a controlled asynchronous Firecrawl site crawlMap and crawl an authorized site with strict scope, verified webhooks, deduplication, and ingestion checks.
↓Workflow 01
Scrape a known page safely with Firecrawl
Extract useful page content while controlling freshness, formats, browser behavior, caching, and retention.
Outcome: Produce a validated page record without leaking credentials or treating arbitrary web text as trusted instructions.
- 01
Confirm the URL may be accessed and define required format, main-content scope, freshness, location, timeout, and storage policy.
- 02
Call Scrape server-side with the key in a secret store, requesting only necessary formats and leaving TLS verification intact.
- 03
Avoid sending cookies or headers unless authorized, disable caching for sensitive work, and use zero-retention or lockdown where available.
- 04
Validate final URL, metadata, content type, extraction completeness, prompt injection, personal data, and expected schema before use.
- 05
Store a timestamped source record and hash, handle errors and stale cache explicitly, and remove browser sessions when interaction ends.
What good use looks like
- Request the minimum output formats.
- Treat extracted text as untrusted.
- Do not send session credentials casually.
Workflow 02
Build a controlled asynchronous Firecrawl site crawl
Map and crawl an authorized site with strict scope, verified webhooks, deduplication, and ingestion checks.
Outcome: Create a bounded website corpus whose source, lifecycle, and failures remain traceable.
- 01
Define allowed domains, paths, query handling, sitemap use, page and depth limits, data policy, refresh cadence, and stop conditions.
- 02
Run Map first to inspect scope, then start a narrowly filtered crawl and persist its job ID and configuration.
- 03
Configure an HMAC secret, verify each raw webhook signature, deduplicate by webhook ID, and handle retries and out-of-order events.
- 04
Validate each page for canonical URL, status, duplicates, extraction errors, unsafe instructions, sensitive data, and indexing eligibility.
- 05
Reconcile the completed job with retrieved results, record omissions and errors, then test update, deletion, cancellation, and cost controls.
What good use looks like
- Inspect a map before crawling.
- Verify webhook signatures.
- Reconcile completion against stored pages.
Official references
Check the current product documentation.
Features, plan limits, availability, and data controls change. These official pages are the starting points used for this guide.