AutoCoder is a suite of large language models designed for code generation tasks, offering superior performance on benchmarks like HumanEval compared to leading proprietary models. It targets developers and researchers seeking advanced code generation capabilities, with a key benefit being its integrated, intelligent code interpreter that can automatically manage package installations and code execution.

How It Works

AutoCoder models are fine-tuned on extensive code datasets, with specific versions based on CodeQwen1.5 and DeepSeek Coder. A distinguishing feature is its code interpreter, which, unlike some competitors, can automatically install necessary Python packages and execute code iteratively until it functions correctly. This approach enhances the model's utility for tasks requiring external libraries or complex execution environments.

Quick Start & Requirements

• Install: Create a conda environment (conda create -n AutoCoder python=3.11), activate it (conda activate AutoCoder), and install dependencies (pip install -r requirements.txt).
• Prerequisites: Python 3.11, Conda. Linux is suggested for deployment.
• Testing: Navigate to the Evaluation directory and run python test_humaneval.py for HumanEval benchmarks, python postprocess_mbpp.py for MBPP, and python test_ds1000.py for DS-1000.
• Web Demo: Install Gradio (pip install gradio==3.48.0) and run python chatbot.py from the /Web_demo directory.
• Resources: Specific hardware requirements are not detailed, but model sizes range from 6.7B to 33B parameters.

Highlighted Details

• Achieves 90.9% accuracy on the HumanEval base dataset, surpassing GPT-4 Turbo (90.2%).
• Demonstrates 82.5% accuracy on the MBPP benchmark.
• Features an intelligent code interpreter capable of automatic package installation and execution.
• Offers models in various sizes (6.7B, 7B, 33B) based on different foundational models.

Maintenance & Community

• The project is maintained by bin123apple.
• Contact is available via email at leib2765@gmail.com or by raising an issue on the repository.
• The project acknowledges guidance from the OpenCodeInterpreter team.

Licensing & Compatibility

• The README does not explicitly state a license. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project's licensing is not clearly defined, which may impact commercial adoption. While performance on benchmarks is highlighted, real-world usability and robustness of the automatic code execution feature require further evaluation.