llm-council  by karpathy

A multi-LLM collaborative framework for enhanced question answering

Created 1 month ago
13,049 stars

Top 3.8% on SourcePulse

GitHubView on GitHub
Project Summary

<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> The llm-council project addresses the challenge of obtaining comprehensive and reliable answers from multiple Large Language Models (LLMs) by orchestrating them into a collaborative council. It targets users who want to leverage the strengths of various LLMs and gain deeper insights through peer review, offering a local web application that aggregates, reviews, and synthesizes LLM responses. The benefit lies in a more robust and insightful final answer derived from collective intelligence.

How It Works

The core approach involves a three-stage process. First, a user query is sent to multiple LLMs simultaneously for initial responses, which are presented side-by-side. Second, each LLM anonymously reviews and ranks the other LLMs' outputs for accuracy and insight. Finally, a designated "Chairman" LLM synthesizes all reviewed responses into a single, consolidated answer. This multi-stage review and synthesis process aims to improve response quality by leveraging diverse perspectives and peer critique.

Quick Start & Requirements

  • Installation: Uses uv for Python dependency management (uv sync) and npm for frontend (cd frontend && npm install).
  • Prerequisites: Requires an OpenRouter API key (OPENROUTER_API_KEY=sk-or-v1-...). Python 3.10+ is needed for the backend.
  • Configuration: Models can be customized in backend/config.py (e.g., COUNCIL_MODELS, CHAIRMAN_MODEL).
  • Running: Execute ./start.sh or run backend (uv run python -m backend.main) and frontend (cd frontend && npm run dev) separately. Access via http://localhost:5173.
  • Tech Stack: Backend: FastAPI, async httpx, OpenRouter API. Frontend: React, Vite, react-markdown. Storage: JSON files. Package Management: uv, npm.

Highlighted Details

  • Orchestrates multiple LLMs (OpenAI, Google, Anthropic, xAI) into a "council" for collaborative querying.
  • Implements a peer-review system where LLMs anonymously critique each other's responses.
  • Features a "Chairman" LLM to synthesize final answers from reviewed outputs.
  • Provides a local web app interface similar to ChatGPT for interacting with the LLM council.

Maintenance & Community

This project was explicitly "vibe coded as a fun Saturday hack" and the author states, "I'm not going to support it in any way, it's provided here as is for other people's inspiration." No community links or roadmap are provided.

Licensing & Compatibility

The README does not specify a software license. Compatibility for commercial use or closed-source linking is undetermined due to the lack of a license.

Limitations & Caveats

The project is provided "as is" with no support intended by the author. It is a "fun Saturday hack" and not intended for production use or further development by the original creator. Users are encouraged to modify the code themselves.

Health Check
Last Commit

1 month ago

Responsiveness

Inactive

Pull Requests (30d)
28
Issues (30d)
13
Star History
2,075 stars in the last 30 days

Explore Similar Projects

Starred by Nir Gazit Nir Gazit(Cofounder of Traceloop), Jared Palmer Jared Palmer(SVP at GitHub; Founder of Turborepo; Author of Formik, TSDX), and
3 more.

haven by redotvideo

0.3%
348
LLM fine-tuning and evaluation platform
Created 2 years ago
Updated 1 year ago
Feedback? Help us improve.