This project provides an AI-powered search and chat interface for the "Wait But Why" blog, enabling users to query and discuss Tim Urban's articles. It's designed for fans of the blog who want to explore its content through natural language.

How It Works

The system leverages OpenAI Embeddings (text-embedding-ada-002) to create vector representations of text chunks from the blog's essays. A search query is also embedded, and cosine similarity is used to find the most relevant passages stored in a Supabase PostgreSQL database with the pgvector extension. The chat functionality builds upon these search results by feeding them into GPT-3.5-turbo to generate contextually relevant answers.

Quick Start & Requirements

• Install dependencies: npm i
• Set up environment variables: OPENAI_API_KEY, NEXT_PUBLIC_SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY in .env.local.
• Scrape data: npm run scrape
• Generate embeddings and store in database: npm run embed (takes 20-30 minutes).
• Run the app: npm run dev
• Requires an OpenAI API key and a Supabase account.

Highlighted Details

• Utilizes cosine similarity for semantic search.
• Stores embeddings in a PostgreSQL database with the pgvector extension.
• Chat interface uses search results to provide context to GPT-3.5-turbo.
• Scraped data is saved to a JSON file before embedding.

Maintenance & Community

• Project creator is active on Twitter for questions.
• The project prioritizes simplicity over composability, with most logic in the homepage component.

Licensing & Compatibility

• The repository does not explicitly state a license.

Limitations & Caveats

• The project sacrifices composability for simplicity, potentially impacting modularity and reusability.
• Requires specific API keys and a Supabase setup, which might be a barrier for some users.