Research map / Large language models / Alignment and preferences
Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Copilot Arena, a VSCode extension that collects pairwise user preferences over inline code completions from 10 LLMs in the wild, with a latency-aware sampling strategy and a FiM prompting scheme, and compares the resulting leaderboard to static and chat-based benchmarks.
Problem
Existing code LLM evaluations rely on static, interview-style benchmarks or chat-based human preference arenas, which lack real users, real tasks, and realistic IDE environments for coding assistance.
Approach
Embed a pairwise preference interface directly into the developer's IDE (Git-diff-style stacked completions, accept-as-vote), sample model pairs with a latency-optimized distribution, and prompt chat models to perform fill-in-the-middle via repeated prefix/suffix and snippet post-processing.
Key results
- Copilot Arena collected over 11,604 pairwise votes from 1,642 users across 10 models, serving over 4.5 million suggestions over 3 months.
- Copilot Arena leaderboard correlates most with Chatbot Arena (coding) at Spearman r_s=0.62 but has r_s<=0.1 with most static benchmarks.
- The latency-aware sampling reduced median pairwise latency by 33% (1.61 to 1.07 seconds) versus uniform sampling; the prompting scheme improved FiM pass@1 in 93% of conditions (e.g., Llama-3.1-70B from 7.4% to 49%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Snip-It prompting (repeat prefix/suffix, then strip duplicated code) added to chat models for FiM (ablation) | chat models on HumanEval-infilling · HumanEval-infilling | pass@1: improved in 93% of conditions; Claude-3.5-Sonnet 56.1% to 73.0% | Helped |
| Latency-optimized model-pair sampling vs uniform sampling | Copilot Arena pair sampling · Copilot Arena deployment | median experienced latency (s): 1.61 to 1.07 seconds (33% reduction) | Helped |
| Copilot Arena leaderboard ranking compared to static benchmarks | model rankings · LiveBench, LiveCodeBench, BigCodeBench | Spearman correlation: r_s ≤ 0.1 | Hurt |
| Task type split (frontend/backend vs algorithm design) | model win-rates · Copilot Arena user preference | significant win-rate changes: 28 of 90 pairwise changes (31.1%) | Mixed |
| Programming language split (non-Python vs Python) | model win-rates · Copilot Arena user preference | significant win-rate changes: 6 of 90 (6.6%) | No clear effect |
| Training-for-infilling (DeepSeek Coder, Codestral, Qwen-2.5 Coder) vs chat-style prompting | FiM-trained code models on Copilot Arena · Copilot Arena user preference | win-rate change | No clear effect |
| Style control (latency, response length) added to Bradley-Terry model (ablation) | Copilot Arena leaderboard · Copilot Arena user preference | model tiers: minimal changes to tiers; only middle-tier swaps | No clear effect |
Surprising findings
- Models explicitly trained for infilling (DeepSeek Coder, Codestral, Qwen-2.5 Coder) did not show large win-rate gains from FiM relative to other models.
- Users selected the first (top) completion 86% of the time, indicating a strong position bias.
Limitations
- Results may not generalize to all real-world coding use cases; pairwise completions and added latency differ from GitHub Copilot.
- GitHub Copilot could not be included because its underlying model is not available via API.
- Only a hand-curated subset of code contexts is released due to privacy concerns.
Experimental setup
Datasets: HumanEval-infilling
Benchmarks: Copilot Arena leaderboard HumanEval-infilling Chatbot Arena (general) Chatbot Arena (code) LiveBench LiveCodeBench BigCodeBench
Metrics: Bradley-Terry strength Spearman rank correlation pass@1 win-rate difference median latency
Baselines: Chatbot Arena (general) Chatbot Arena (code) LiveBench LiveCodeBench BigCodeBench
Builds on: Chatbot Arena Bradley-Terry model HumanEval-infilling Fill-in-the-middle
Closest papers by meaning
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.88)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.88)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.88)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.87)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.