DeepSeek is an experimental, LLM-powered retrieval engine designed to comprehensively collect and enrich entities from a vast number of internet sources. Unlike typical "answer engines" that aim for a single correct response, DeepSeek functions as a "retrieval engine," outputting a detailed table of entities and their associated data, complete with confidence scores. This makes it suitable for users needing exhaustive data aggregation rather than concise summaries.

How It Works

DeepSeek employs a multi-step "flow engineering" architecture. It begins with a "Plan" phase, where the LLM defines the entities to extract and the relevant data columns based on the user query. The "Search" phase utilizes both keyword and neural search via Exa to find relevant content. In the "Extract" phase, a novel technique inserts special tokens into content, allowing the LLM to efficiently identify and extract specific entities and their associated data. Finally, the "Enrich" phase uses a smaller LLM to populate the defined columns for each entity, assigning confidence scores to the extracted data.

Quick Start & Requirements

• Install via npm, yarn, pnpm, or bun.
• Run npm run dev (or equivalent) to start the dev server.
• Requires API keys for Anthropic and Exa, configured in a .env file.
• Running the agent can take ~5 minutes and cost $0.1-$3 in API credits.
• Demo: https://deep-seek.vercel.app/

Highlighted Details

• Processes hundreds of sources to retrieve and enrich dozens of entities.
• Generates confidence scores for extracted data, highlighting potential conflicts or guesses.
• Utilizes Exa for both keyword and neural search capabilities.
• Employs a token-efficient LLM extraction technique using special sentence delimiters.

Maintenance & Community

• Project lead can be contacted via email (david@aomni.com) or Twitter for collaboration and discussion.
• Future work includes sorting/ranking, improved entity resolution, source verification, deep browsing, and streaming data.

Licensing & Compatibility

• The README does not explicitly state a license.

Limitations & Caveats

• The project is experimental and the provided demo does not run the agent due to cost.
• Entity resolution for similar items (e.g., M2 vs. M3 Macbooks) needs improvement.
• Source verification during enrichment is an area for enhancement.
• Real-time streaming of results is not yet implemented.