Research map / Large language models / Agents, code and math
CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CodeAssistBench (CAB), an automated pipeline that builds multi-turn, project-grounded programming assistance benchmark from GitHub question issues with Dockerized environments, simulated users, and LLM judging.
Problem
Existing programming assistant benchmarks are single-turn, built on isolated snippets or Stack Overflow Q&A, and require manual curation, failing to evaluate multi-turn assistance in complete project environments.
Approach
Automatically converts GitHub issues labeled as questions into multi-turn assistance scenarios with containerized build environments, LLM-extracted satisfaction conditions, and BM25-retrieved simulated user replies, evaluated by a multi-agent user/maintainer/judge loop.
Key results
- Best correctness on recent (post-cutoff) repositories was 16.49% for ChatGPT 4.1 Mini, with Haiku 3.5 lowest at 7.22%.
- The LLM judge reached 65.92% average agreement with human experts versus 78.28% inter-human agreement (Cohen's kappa 0.68).
- Benchmark contains 3,286 retained multi-turn issues from 214 repositories across seven languages, with 238 validated Docker environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use all-time repositories rather than post-training-cutoff recent repositories | Evaluated LLMs on CAB · CAB repositories | correctness (%): 7.22–16.49% -> 13.58–29.14% | Helped |
| Evaluate on automatically generated repositories rather than post-training-cutoff real repositories (ablation) | Claude 3.7 Sonnet · Synthetic repositories vs. recent real repositories | correctness (%): 11.34% -> 74% | Helped |
Surprising findings
- Claude 3.7 Sonnet reached 74% correctness on synthetic repositories versus 11.34% on recent real repositories, suggesting the recency gap is driven by post-cutoff framework changes rather than AI-generated code properties.
- Recent repositories achieved a 97.6% Docker build success rate versus 78.2% for All-Time repositories.
Limitations
- Satisfaction condition extraction favors precision over recall (86.3% accurate but only 65.7% complete).
- Evaluation covers a sampled subset of 544 issues and only seven languages; only Docker-buildable issues are included.
- LLM judge agreement with humans (65.92%) is well below inter-human agreement (78.28%), and the simulated user relies on BM25-matched templated replies.
Experimental setup
Datasets: GitHub issues labeled question/help-wanted from 770 repositories
Benchmarks: CodeAssistBench (CAB)
Metrics: correctness (correct / partially correct / incorrect) average CAB turns verbosity judge-human agreement Cohen's kappa
Baselines: ChatGPT 4.1 Mini DeepSeek R1 Llama 3.3 70B Haiku 3.5 Sonnet 3.7 Sonnet 3.7 (Think)
Builds on: SWE-Bench InfiBench StackEval BM25
Closest papers by meaning
- CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules (ICLR 2024 · similarity 0.88)
- InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models (NeurIPS 2024 · similarity 0.88)
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.87)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.86)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.