Copilot Arena is an open-source VS Code extension that provides paired AI coding completions and in-line edits from various state-of-the-art LLMs. It aims to help users discover their preferred coding assistants and contributes to evaluating LLM performance in real-world coding scenarios.

How It Works

The extension integrates directly into VS Code, offering two primary features: autocomplete and in-line editing. For autocomplete, it displays two LLM-generated completions side-by-side, allowing users to accept either with a key press. In-line editing allows users to highlight code, provide a prompt, and receive two diffs of the suggested changes, which can then be accepted or rejected. This paired-response approach facilitates direct comparison and user preference tracking.

Quick Start & Requirements

• Install the Copilot Arena extension from the Visual Studio Code Marketplace.
• Disable any other AI completion providers (e.g., GitHub Copilot) before using Copilot Arena.
• Optional: Create a username via the extension's sidebar icon to track personal preferences and contribute to future leaderboards.
• Official documentation and FAQs are available via links in the README.

Highlighted Details

• Supports multiple LLMs including GPT-4o, Codestral, and Llama-3.1.
• Features personal leaderboards to track user preferences for different models.
• Allows users to opt-out of data collection for privacy.
• Backend code is open-sourced, welcoming community contributions.

Maintenance & Community

• Active community engagement is encouraged via Discord and GitHub issues.
• The project has a citation available for academic use.

Licensing & Compatibility

• The project is open-source, with the backend code specifically mentioned as open-sourced. The specific license for the extension itself is not explicitly stated in the provided text, but it is described as "free to use."

Limitations & Caveats

Code from the current file is sent to servers for completion generation, and this data transfer cannot be disabled. Users working with sensitive information should opt-out of data collection. A global leaderboard is planned but not yet implemented.