Research map / Large language models / Agents, code and math
StackEval: Benchmarking LLMs in Coding Assistance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StackEval and StackUnseen, Stack Overflow-derived coding-assistance benchmarks (plus a human-labelled LLM-as-judge dataset) and evaluates many LLMs as solvers and judges, including self-preference tests.
Problem
Existing coding benchmarks (HumanEval, SWE-bench, MultiPL-E) focus on function completion or narrow tasks, are prone to training-data leakage, and do not cover broad coding-assistance tasks like debugging, implementation, optimization and conceptual questions; reliable open-ended evaluation is also
Approach
Curates real Stack Exchange questions with accepted answers into a multi-language, multi-task benchmark, adds a time-separated "unseen" split of recent questions to limit leakage, and scores answers with an LLM judge using a reference answer and a 0-3 acceptability rubric, validated against human labels.
Key results
- O1 Preview achieved 95.5% acceptance on StackEval and 83.0% on StackUnseen, the highest among evaluated models.
- Including a reference answer improved GPT-4 Turbo judge accuracy from 78.3% to 83.6%, and CoT plus reference reached 84.4% against human labels.
- Acceptance rates dropped substantially on StackUnseen for most models (e.g., GPT-4o 83.0% to 55.7%, Llama3.1-405B 76.5% to 47.9%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a reference answer to the judging prompt (ablation) | LLM judges · LLM-as-a-Judge benchmark | accuracy: 78.3% $\pm$ 1.9% -> 83.6% $\pm$ 0.4%; 81.3% $\pm$ 0.3% -> 82.1% $\pm$ 1.3%; 80.7% $\pm$ 0.2% -> 81.9% $\pm$ 1.6%; 78.7% | Helped |
| Add chain-of-thought reasoning without a reference answer (ablation) | LLM judges · LLM-as-a-Judge benchmark | accuracy: 78.3% $\pm$ 1.9% -> 73.9% $\pm$ 1.2%; 80.7% $\pm$ 0.2% -> 77.6% $\pm$ 0.2%; 78.7% $\pm$ 0.4% -> 76.1% $\pm$ 0.4%; 71.3% | Hurt |
| Add chain-of-thought reasoning and a reference answer (ablation) | LLM judges · LLM-as-a-Judge benchmark | accuracy: 78.3% $\pm$ 1.9% -> 84.4% $\pm$ 1.8%; 81.3% $\pm$ 0.3% -> 82.8% $\pm$ 0.2%; 78.7% $\pm$ 0.4% -> 81.2% $\pm$ 0.9%; 71.3% | Helped |
| Add chain-of-thought reasoning and a reference answer (ablation) | LLM judges · LLM-as-a-Judge benchmark | accuracy: 80.7% $\pm$ 0.2% -> 81.5% $\pm$ 0.3%; 79.9% $\pm$ 1.2% -> 80.5% $\pm$ 0.2% | Mixed |
| Evaluate recent, unseen Stack Overflow questions instead of historical questions | LLMs on coding assistance tasks · StackEval vs. StackUnseen | acceptance rate: 95.5% -> 83.0%; 89.5% -> 76.3%; 90.7% -> 71.6%; 86.9% -> 66.5%; 81.9% -> 52.6%; 80.2% -> 50.5%; 76.5% -> 47.9% | Hurt |
Surprising findings
- Chain-of-thought alone did not improve judge accuracy and sometimes reduced it.
- With reference answers, no tested judge showed significant self-preference (all p > 0.05), contrary to prior summarization findings.
Limitations
- Benchmark derived from Stack Overflow inherits community and voting biases and may not cover all paradigms.
- Evaluates one-shot answers and does not test agents that run or verify code; efficiency and maintainability not assessed.
- LLM judges may share blind spots with evaluated models; judge study is based on a small 136-question set.
Experimental setup
Datasets: Stack Exchange data dumps (Stack Overflow, Ask Ubuntu, Super User, Unix & Linux, Software Engineering, Server Fault)
Benchmarks: StackEval StackUnseen LLM-as-a-Judge benchmark
Metrics: acceptance rate judge accuracy Wilcoxon signed-rank p-value (self-preference)
Baselines: GPT-4 Turbo Claude-3.5 Sonnet Gemini-1.5 Pro Mistral Large 2 WizardLM-2 8x22B Llama3.1-70B O1 Preview
Builds on: LLM-as-a-Judge MT-Bench HumanEval MultiPL-E
Closest papers by meaning
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.91)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.90)
- Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks (ICML 2024 · similarity 0.90)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.90)
- Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.