This repository is a curated list of state-of-the-art Large Language Models (LLMs) specifically designed for code-related tasks, serving researchers and developers in the field of AI for code. It provides a comprehensive overview of top-performing models, evaluation toolkits, leaderboards, and seminal research papers, aiming to track advancements and facilitate informed adoption decisions.

How It Works

The project functions as a living document, aggregating and categorizing resources related to code LLMs. It highlights models based on performance metrics like HumanEval and MBPP, links to evaluation frameworks (e.g., bigcode-evaluation-harness, code-eval), and compiles extensive lists of papers covering pre-training, instruction-tuning, alignment, prompting, and benchmarking. The structure allows users to quickly compare model capabilities and explore the research landscape.

Quick Start & Requirements

This repository is a curated list and does not require installation or execution. It serves as a reference guide.

Highlighted Details

• Features a dynamic leaderboard ranking code LLMs by HumanEval and MBPP scores, with recent additions like Qwen2.5-Coder and OpenCoder.
• Compiles a vast collection of papers, including technical reports and preprints, on code LLM development and evaluation.
• Lists and describes various evaluation toolkits and benchmarks crucial for assessing code generation capabilities.
• Includes links to model sources (GitHub, Hugging Face) and associated research papers for deeper dives.

Maintenance & Community

The repository is actively maintained, with recent updates in November 2024 reflecting new model releases. Contributions are welcomed via pull requests. Contact information for the primary author is provided.

Licensing & Compatibility

The repository itself is licensed under an unspecified license, but it primarily links to external resources which have their own licenses. Users should verify the licensing of individual models and datasets.

Limitations & Caveats

As a curated list, the "best" models are subjective and based on the metrics presented. The rapidly evolving nature of LLMs means the leaderboard and paper lists may require frequent updates to remain current.