<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> InterCode provides a lightweight, flexible framework for building and benchmarking interactive code environments, specifically designed to evaluate language agents capable of coding. It standardizes interactive coding tasks with execution feedback, benefiting researchers and developers in this domain.

How It Works

The framework leverages Docker to create isolated, interactive environments for tasks like Bash, SQL, Python, and CTF. It focuses on providing execution feedback, enabling standardized evaluation of code-generating language agents through a flexible and easy-to-use API.

Quick Start & Requirements

Installation is available via PyPI (pip install intercode-bench) or by building from source (clone repo, conda env create -f environment.yml, python run_demo.py). Prerequisites include Python >= 3.8 and a running Docker daemon. Example interaction scripts for Bash and SQL are provided.

Highlighted Details

• Supports diverse interactive coding environments: Bash, CTF, Python, SQL.
• Facilitates standardized benchmarking of language agents with execution feedback.
• Developed under the NeurIPS 2023 D&B track.
• Offers an interactive CLI for task execution.

Maintenance & Community

Developed by Princeton NLP researchers (Yang, Prabhakar, Narasimhan, Yao). Welcomes community contributions via pull requests and issues. No specific community channels (e.g., Discord, Slack) or roadmap are detailed in the README.

Licensing & Compatibility

Released under the MIT license, generally permissive for commercial use and integration with closed-source projects.

Limitations & Caveats

Requires Docker installation and a running daemon. Running experimental scripts necessitates API keys (OpenAI, PaLM) configured via environment variables or a keys.cfg file.