Research map / Large language models / AI and society
CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
Hanjun Luo, Chiming Ni, Jiaheng Wen, Zhimu Huang, Bingduo Liao, Sylvia Chung, Yiran Wang, Yingbin Jin, Jialin Li, Xinfeng Li, Wenyuan Xu, XiaoFeng Wang and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Copilot (Claude-Sonnet-4) assistance to human developers | human developers · CentaurEval overall | Overall Pass@1: 18.89 -> 31.11 | Improved |
| add human collaboration to the autonomous agent | Claude-Sonnet-4 Copilot · CentaurEval overall | Overall Pass@1: 0.67 -> 31.11 | Improved |
| add minimal procedural human intervention (C1 vs C0) (ablation) | Claude-Sonnet-4 · CentaurEval overall | Overall Pass@1: 0.67 -> 2.89 | Improved |
| add minimal procedural intervention (C1 vs C0) (ablation) | GPT-4o · CentaurEval overall | Overall Pass@1: 0.00 -> 0.00 | No clear effect |
| add minimal procedural intervention (C1 vs C0) (ablation) | all five LLMs · CentaurEval overall | Partial Pass@1: Claude-Sonnet-4 19.24 -> 30.13 | Improved |
| sample 10 attempts instead of 1 (ablation) | Claude-Sonnet-4 C0 · CentaurEval overall | Overall Pass: 0.67 -> 3.11 | Improved |
| increase task difficulty from easy to hard (ablation) | human-only · CentaurEval | Pass: 36.7 -> 6.7 | Hurt |
| increase task difficulty from easy to hard (ablation) | human-AI collaboration · CentaurEval | Pass: 43.3 -> 23.3 | Mixed |
| add AI collaboration to humans on completion time | human developers · CentaurEval overall | completion time (s): 3203 -> 2791 | Improved |
Closest papers by meaning
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.89)
- Huxley-G\"odel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine (ICLR 2026 · similarity 0.89)
- CORVUS: Context Optimization and Reduction Via Underlying Synchronization for LLM Coding Agents (NeurIPS 2026 · similarity 0.88)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.88)
- From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.88)
- The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
