us/crw: Rust-native Web Scraper & Crawler for AI Agents

This project offers a high-performance, low-resource Rust-native alternative to web scraping and crawling tools like Firecrawl and Tavily. It provides a drop-in compatible API for AI agents and developers, enabling faster data acquisition with a significantly reduced memory footprint. The core benefit is a single, static binary that simplifies deployment and operation, whether self-hosted or via a managed cloud service.

How It Works

Built entirely in Rust, us/crw eschews traditional Node.js or Python-based stacks, eliminating dependencies like Redis or headless browser sidecars from the critical request path. It exposes a native /v1 API for scraping, crawling, mapping, and searching, alongside a /v2 compatibility layer for seamless migration from Firecrawl. This architecture allows for sub-second cold starts and a bounded memory ceiling, making it exceptionally efficient for agent-driven workflows where latency compounds.

Quick Start & Requirements

• Primary Install: Docker (docker run -p 3000:3000 ghcr.io/us/crw), pip install crw (Python SDK), npx crw-mcp (Node.js MCP server), brew install us/crw/crw, or via a one-line shell script (curl -fsSL https://fastcrw.com/install | sh).
• Prerequisites: None beyond standard OS/hardware. Python 3.10+ required for the Python SDK.
• Resource Footprint: ~50 MB RAM idle, single binary (~8 MB).
• Links: Documentation (docs.fastcrw.com), Benchmarks (fastcrw.com/benchmarks), Managed API (api.fastcrw.com).

Highlighted Details

• Performance: Benchmarked 2.3x faster than Tavily and 1.5x faster than Firecrawl on 1K-URL tests, achieving 63.74% truth-recall on a public dataset.
• Firecrawl Compatibility: Offers a /v2 API surface for direct migration of existing Firecrawl v2 SDK projects.
• MCP Server: Includes a built-in MCP (Model Context Protocol) server, enabling direct integration with AI agents like Claude Code, Cursor, and Gemini CLI without custom glue code.
• Change Tracking: Features a primitive for diffing web pages against prior snapshots, with optional LLM-based analysis for "meaningful change" detection.
• Agent Skills: Provides pre-packaged skills for AI agents to perform complex web interactions like scraping, crawling, and searching.

Maintenance & Community

The project maintains an active presence with a Discord server (discord.gg/kkFh2SC8) and an X/Twitter handle (@fast_crw). Contributions are welcomed via GitHub pull requests.

Licensing & Compatibility

The open-source core is licensed under AGPL-3.0. For closed-source products or hosted services that cannot comply with AGPL's source-availability requirements, a commercial carve-out is available through the managed api.fastcrw.com offering or via direct commercial licensing by contacting hello@fastcrw.com.

Limitations & Caveats

Self-hosted deployments exposed to third parties are subject to AGPL-3.0 obligations. The managed service offers an AGPL carve-out but incurs costs. While benchmarks show strong performance, real-world results may vary based on network conditions and target website complexity. The "fast mode" prioritizes latency over maximum recall.