This project addresses the challenge of AI agents struggling to parse modern, JavaScript-heavy developer documentation sites. It provides a high-performance web service that converts these dynamic sites into clean, LLM-optimized Markdown, making technical information accessible to AI agents. The target audience includes developers, researchers, and anyone building AI systems that need to comprehend technical documentation, offering a significant benefit in improving AI's ability to process and utilize complex documentation.

How It Works

The core approach leverages Firecrawl's headless browser to execute JavaScript and capture fully rendered HTML content. This dynamic content is then transformed into clean, semantic Markdown. The system intelligently removes extraneous elements like navigation, boilerplate, and duplicate content to conserve AI context tokens. Key architectural choices include parallel URL processing for efficiency, a Redis-backed caching layer for reduced API calls, and configurable content filtering strategies to tailor the output for AI consumption.

Quick Start & Requirements

• Primary install/run command: Clone the repository, run npm install, create a .env.local file with your FIRECRAWL_API_KEY, and start the development server with npm run dev.
• Prerequisites: Node.js 20+, npm or yarn, and a Firecrawl API key. Redis is recommended for production caching.
• Deployment: Easily deployable to Vercel.
• Demo: A live demo is available at llm.codes.

Highlighted Details

• Parallel Processing: Fetches up to 20 URLs concurrently using batched promises.
• Smart Caching: Utilizes a Redis-backed 30-day cache to minimize API calls and improve response times.
• Content Filtering: Offers multiple strategies to remove navigation, boilerplate, platform availability strings (e.g., "iOS 14.0+"), and duplicate content.
• Pattern-Based Site Support: Automatically supports many documentation sites through intelligent URL pattern matching (e.g., docs.*, developer.*, *.github.io) and explicit exceptions.

Maintenance & Community

Contributions are welcomed via pull requests. The project is hosted on GitHub. No specific community channels (like Discord or Slack) or notable contributors/sponsorships are detailed in the README.

Licensing & Compatibility

The project is licensed under the MIT License, which is generally permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

Browser notifications require user permission and may not function in all browsers or configurations. Without Redis caching, users might encounter API rate limits. While pattern matching covers many sites, some documentation structures may require explicit exceptions or community-reported issues for full support.