Firecrawl

Firecrawl is an open-source web crawler and scraper API. This template follows the official self-hosting guide (release v2.11.162): PostgreSQL queue, no database authentication, no FoundationDB, no queue-admin UI unless you keep the generated BULL_AUTH_KEY.

The stack:

• ui — firecrawl-ui playground (scrape, crawl, extract, map, search)
• api — harness.js --start-docker (API server plus worker, extract-worker, and NUQ queue workers in one container)
• playwright — Chromium microservice for JS-rendered pages
• redis — cache and rate-limit store (no password, matching upstream Compose)
• rabbitmq — NUQ task broker
• postgres — nuq-postgres (Postgres 17 + pg_cron + NUQ schema). Database name is postgres because pg_cron is compiled against that database.

Resource recommendation

Upstream Compose sizes the API at 4 vCPU / 8 GB RAM and Playwright at 2 vCPU / 4 GB RAM. Deploy on a Pro plan or a dedicated server.

Configuration

The API domain and UI domain are required. Authentication is off (USE_DB_AUTHENTICATION=false); requests do not need an API key. Do not expose this baseline to untrusted networks without a reverse-proxy ACL or auth layer.

Optional LLM features (JSON extract, summary, /v2/extract): set OPENAI_API_KEY and optionally OPENAI_BASE_URL at deploy time. Any OpenAI-compatible provider works (OpenAI, OpenRouter, xAI, Ollama, Zeabur AI Hub). Leave both blank to skip LLM. You can still change MODEL_NAME on the api env tab after deploy.

BULL_AUTH_KEY is auto-generated. Find it on the api service env tab.

Quick test

After the API is up, run a real scrape (this is the functional smoke test, not the readiness heartbeat):

curl --fail-with-body --max-time 75 \
  -X POST https://<your-domain>/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"],"timeout":60000}'

A successful body looks like {"success":true,"data":{"markdown":"...","metadata":{"statusCode":200}}}.

Self-hosted feature support

Need	This template
Core scrape, crawl, map, search	Included (Fetch + Playwright)
LLM extract / summary	Add an OpenAI-compatible key as above
Fire-engine anti-bot, screenshots, page actions	Not included; those paths need Fire-engine / Cloud
Agent, Browser, interact, product/audio/video formats	Use Firecrawl Cloud or extra external services

Playground UI

Open the UI domain after deploy. Nginx on the UI service proxies /v2, /v1, and /v0 to the API, so the playground talks same-origin and does not need CORS. If the API URL is empty, open Settings (gear) and paste the UI URL (or the API URL). Config is stored in the browser (localStorage).

Operators can also open Bull Board at:

https://<api-domain>/admin/<BULL_AUTH_KEY>/queues

Versioning

Upstream GHCR images only publish :latest. This template pins each Firecrawl image by digest (pulled 2026-09-02) so deploys stay reproducible. Redis and RabbitMQ use upstream tags.

Reference

• Self-host docs: https://docs.firecrawl.dev/contributing/self-host
• GitHub: https://github.com/firecrawl/firecrawl
• Open Source vs Cloud: https://docs.firecrawl.dev/contributing/open-source-or-cloud
• Playground UI: https://github.com/obeone/firecrawl-ui
• License: AGPL-3.0 (Firecrawl); MIT (firecrawl-ui)