BigSet addresses the challenge of creating and maintaining structured datasets from the live web by automating the entire process from data description to scheduled refreshes. It is designed for users who need up-to-date, verified data without the complexity of traditional web scraping, offering a significant benefit in time and effort savings.

How It Works

BigSet utilizes a natural language interface where users describe the data they need. An AI infers the schema, and autonomous agents then conduct web research, fetch real data, verify its accuracy, and deduplicate entries to produce a structured table. This approach bypasses the need for manual scraper configuration, allowing agents to handle data discovery and maintenance automatically.

Quick Start & Requirements

• Primary install/run command: Clone the repository, copy .env.example to .env, and run make dev.
• Non-default prerequisites: Docker, Make, API keys for TinyFish (web search/fetching), OpenRouter (LLM calls), and Clerk (user authentication). Requires self-hosted Postgres and Convex.
• Resource footprint: make dev starts multiple Docker services including frontend, backend, database, and workflow inspector. Setup involves obtaining API keys and configuring the .env file.
• Links: TinyFish API Keys: tinyfish.ai/api-keys, OpenRouter API Keys: openrouter.ai/settings/keys, Clerk Dashboard: dashboard.clerk.com.
• Free tier: Limited to 2,500 row operations per account per calendar month.

Highlighted Details

• Automated dataset generation from plain English descriptions.
• Autonomous agents perform web search, data fetching, verification, and deduplication.
• Configurable scheduled refreshes (30 minutes to weekly) keep datasets current.
• Datasets are downloadable as CSV or XLSX files.

Maintenance & Community

BigSet is being built in the open, with active development and a welcome for feedback and contributions via GitHub Issues. Project updates are shared on Twitter (@Tiny_Fish).

Licensing & Compatibility

The project is licensed under AGPL-3.0. This strong copyleft license may impose restrictions on use within proprietary or closed-source applications, requiring careful legal review for commercial adoption.

Limitations & Caveats

BigSet is experimental, and users should expect rough edges, particularly with schema inference accuracy. It cannot access data behind logins or paywalls. Datasets are currently export-only; direct SQL querying is planned for a future release. Dataset generation typically takes 2-5 minutes.