Research map / Large language models / Agents, code and math
RefineBench: Evaluating Refinement Capability in Language Models
Young-Jun Lee, Seungone Kim, Byung-Kwan Lee, Minkyeong Moon, Yechan Hwang, Jong Myoung Kim, Graham Neubig, Sean Welleck, Ho-Jin Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add full checklist as evaluation criteria to self-refinement (ablation) | Gemini 2.5 Pro · RefineBench | Pass@5: 31.3 -> 75.8 | Improved |
| add full checklist as evaluation criteria to self-refinement (ablation) | LLaMA-3.1-70B-Instruct · RefineBench | Pass@5: 4.6 -> 48.2 | Improved |
| add explicit checklist feedback (guided refinement) instead of self-refinement | Claude-Opus-4.1 · RefineBench | Pass@5: 20.8 -> 98.4 | Improved |
| add explicit checklist feedback (guided refinement) | LLaMA-3.1-8B-Instruct · RefineBench | Pass@5: +28.7 | Improved |
| use reasoning (thinking) variant instead of instruct | Qwen3-30B-A3B · RefineBench self-refinement | avg improvement: -1.6 -> +1.4 | Improved |
| use reasoning model o1 instead of GPT-4o | self-refinement · RefineBench | avg improvement: -1.4 -> -0.2 | Improved |
| more self-refinement turns (5 vs 1) | Claude-Sonnet-4 · RefineBench | Pass@t: +0.8 | No clear effect |
| more self-refinement turns (5 vs 1) | DeepSeek-R1 · RefineBench | Pass@t: -0.1 | No clear effect |
| more self-refinement turns (5 vs 1) | DeepSeek-R1 (Qwen-32B) · RefineBench | Pass@t: -2.5 | Hurt |
| provide only 50% of checklist feedback (partial guidance) | LMs · RefineBench | Pass@t | Mixed |
| longer termination turn (continue refining more turns) | reasoning LMs · RefineBench | Pass@5 correlation: p-value < 0.01, R2 = -0.477 | Hurt |
| more reasoning tokens for self-refinement | Gemini 2.5 Pro · RefineBench | Pass@t | No clear effect |
Closest papers by meaning
- RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code (ICLR 2025 · similarity 0.90)
- Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024 · similarity 0.89)
- GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements (ICML 2024 · similarity 0.89)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.89)
- RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation (ICML 2025 · similarity 0.89)
- Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
