Summary

Searcharvester provides a self-hosted solution for AI agents, combining a Tavily-compatible search API with a robust URL-to-Markdown content extraction service. It offers a deep research agent capable of generating cited reports, eliminating the need for API keys and quotas, and granting users full control over their data and compute.

How It Works

The system comprises three core HTTP services deployed via Docker Compose: /search leverages SearXNG for access to over 100 search engines, offering Tavily API compatibility. /extract uses trafilatura to convert web page URLs into clean Markdown, featuring size presets and pagination for efficient content handling. The /research service acts as an orchestrator, spawning ephemeral LLM-agnostic agents that utilize search and extract capabilities to synthesize comprehensive, cited Markdown reports based on user queries.

Quick Start & Requirements

Installation is streamlined via Docker Compose. After cloning the repository and configuring config.yaml, users can start the services with docker compose up -d. The /research endpoint requires LLM credentials (e.g., OpenAI API key and base URL) to be configured in a .env file. Testing commands for search, extract, and research are provided in the README.

Highlighted Details

• Offers drop-in replacements for Tavily's /search and /extract APIs.
• trafilatura integration provides clean Markdown extraction with configurable size presets (s, m, l, f) and efficient pagination for long documents.
• The /research agent employs a plan-gather-synthesize methodology, producing cited Markdown reports.
• Fully LLM-agnostic, supporting any OpenAI-compatible endpoint (OpenAI, vLLM, Ollama, etc.).
• No API keys or quotas are necessary for core search and extraction functionalities.

Maintenance & Community

Specific details regarding maintainers, community channels (like Discord/Slack), or roadmap are not explicitly detailed in the provided README.

Licensing & Compatibility

The project's own code is licensed under MIT, promoting broad compatibility for commercial use. However, upstream SearXNG components are subject to the AGPL license, which may impose obligations if those specific components are modified and redistributed.

Limitations & Caveats

The /research functionality is dependent on external LLM credentials. While the project code is MIT licensed, the AGPL license on certain upstream dependencies warrants careful consideration for commercial distribution scenarios.