AnakinScraper is an open-source, self-hostable web scraping API designed to convert any website into LLM-ready Markdown or structured JSON. It targets developers and power users building RAG pipelines, AI agents, or data extraction systems, offering a zero-dependency, single-command deployment and advanced anti-detection capabilities.

How It Works

The core architecture employs a flexible handler chain: it first attempts a direct HTTP fetch, falling back to an anti-detect browser (Camoufox) for JavaScript-heavy or protected sites, and optionally to external APIs. This tiered approach optimizes performance and cost. Smart proxy selection uses Thompson Sampling, a machine learning algorithm, to dynamically choose the best proxy per domain based on real-time success rates. For structured data, it integrates with Google Gemini AI, transforming scraped content into clean JSON.

Quick Start & Requirements

The project offers a zero-configuration, single-command setup via Docker (make up). Alternatively, it can be run as a single Go binary (go run cmd/server/main.go) requiring Go 1.25+. For JavaScript-heavy sites, the Camoufox browser service (run via Docker or Python) is recommended. Optional PostgreSQL is supported for persistent job storage. A React-based web dashboard is included for monitoring and management.

Highlighted Details

• Handler Chain Fallback: Prioritizes efficient HTTP fetching, seamlessly falling back to Camoufox browser or external APIs for complex sites, minimizing costs.
• Camoufox Anti-Detect Browser: Utilizes a Firefox-based browser with realistic fingerprints, offering robust anti-detection compared to standard headless Chrome.
• Thompson Sampling Proxy Selection: Dynamically optimizes proxy usage per domain, learning from success and failure rates.
• Gemini AI Integration: Enables AI-powered structured JSON extraction from scraped web content.
• LLM-Ready Output: Automatically cleans boilerplate, producing Markdown suitable for direct use in Large Language Model pipelines.

Maintenance & Community

The project maintains an active community, with a Discord server available for questions, discussion, and support.

Licensing & Compatibility

AnakinScraper is licensed under AGPL-3.0. This strong copyleft license requires that any modifications or derivative works, if distributed or offered as a hosted service, must also be made open-source under the same license.

Limitations & Caveats

Structured JSON extraction via Gemini AI necessitates providing a Google Gemini API key. While the core server runs without the browser service, JavaScript-heavy or heavily protected websites will likely require the Camoufox browser component for successful scraping. The AGPL-3.0 license imposes significant obligations for commercial or hosted service deployments.