This project provides a web application allowing users to query the open-sourced Twitter algorithm. It targets developers and researchers interested in understanding the algorithm's codebase through a conversational interface. The benefit is an accessible, interactive way to explore complex code.

How It Works

The application features a NextJS/Tailwind CSS frontend and a FastAPI backend. It utilizes a Pinecone vector database to store embeddings of the Twitter algorithm's codebase. Users interact via the frontend, which sends queries to the backend. The backend retrieves relevant code snippets from Pinecone, generates embeddings for them, and then uses an LLM (likely OpenAI's) to answer questions based on these snippets.

Quick Start & Requirements

• Install: Clone the repository, install Node dependencies (npm i), install Python dependencies (pip install -r requirements.txt in backend/).
• Prerequisites: OpenAI API key and organization ID, Pinecone API key. A Pinecone index named pinecone-index with a vector dimension of 1536 is required.
• Setup: Run python create_vector_db.py in backend/ to embed the codebase. Run the backend with uvicorn main:app --reload. The backend URL needs manual configuration for local development.

Highlighted Details

• Interactive querying of the Twitter algorithm codebase.
• NextJS frontend, FastAPI backend, Pinecone vector DB architecture.
• Dockerfile provided for deployment.
• Public instance currently covered by author's OpenAI API costs.

Maintenance & Community

The project author intends to continue development based on demand. Potential improvements are listed, including replacing Pinecone with NumPy, implementing websockets, and improving code splitting and retrieval.

Licensing & Compatibility

The repository does not explicitly state a license.

Limitations & Caveats

The backend URL is hardcoded to the live server and requires manual modification for local execution. The current code splitter is basic; a token-based splitter is suggested as an improvement. The UI is noted as needing significant improvement by the author.