SIA (Self-Improving AI) is a framework designed to autonomously enhance the performance of any AI system, model, or agent on benchmark tasks. It targets researchers and practitioners seeking to automate the optimization of AI systems, offering significant performance gains and runtime reductions across various scientific and machine learning challenges.

How It Works

SIA operates through an iterative, self-improving loop involving three coordinated AI agents. A Meta-Agent initializes a task-specific Target Agent. The Target Agent then attempts the task, logging its actions and results. A Feedback/Improvement Agent reviews these logs, identifies areas for enhancement, and updates the Target Agent's harness and/or weights. This autonomous refinement process allows the system to continuously improve its ability to solve complex scientific tasks.

Quick Start & Requirements

• Installation: Install via pip, choosing a backend: pip install 'sia-agent[claude]' for Claude models, or pip install 'sia-agent[openhands]' for multi-provider support (Gemini, OpenAI, Anthropic, etc.).
• Prerequisites: Python 3.x, and API keys for the chosen LLM providers (e.g., ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENAI_API_KEY).
• Running: Execute with sia --task <task_name> --max_gen <generations> --run_id <id>. Bundled tasks include gpqa, lawbench, longcot-chess, and spaceship-titanic.
• Documentation: Detailed guides are available at docs/configuration.md, docs/walkthrough.md, and docs/troubleshooting.md.

Highlighted Details

• Ranks #1 across all generations on the OpenAI MLE-Bench Hard benchmark.
• Achieved 70.1% Top-1 accuracy on LawBench, significantly surpassing the prior state-of-the-art (45%).
• Delivered a 14x speedup on optimizing the AlphaFold-3 Triangle Multiplicative Update as a Triton kernel.
• Scored 0.289 MSE norm on scRNA-seq Denoising, outperforming the previous SOTA of 0.220.

Maintenance & Community

The provided README does not detail community channels (e.g., Discord, Slack), a public roadmap, or notable maintenance contributors beyond the paper's authors.

Licensing & Compatibility

The README does not explicitly state the project's license. This omission requires further investigation for compatibility with commercial or closed-source projects.

Limitations & Caveats

The README focuses on the framework's capabilities and performance achievements. It does not specify any current limitations, alpha/beta status, or known issues.