Research map / Large language models / Reasoning and chain of thought
Teaching Language Models to Critique via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a critic LLM for code generation with execution-guided SFT followed by GRPO, using the revision success of a fixed generator as reward, to give actionable feedback for iterative critique-revision.
Problem
Iterative self-improvement with LLMs is limited by feedback that is neither accurate in judging correctness nor actionable for fixing errors; self-critique yields little gain in code generation.
Approach
Decouple a specialized critic from the generator and train it with a proxy objective (whether the generator's revision based on the critique becomes correct), bootstrapped by execution-guided critique synthesis (hint-conditioned critiques, distilled via SFT) and optimized with GRPO instead of PPO to reduce credit-assignment variance.
Key results
- On CodeContests with Qwen2.5-Coder generator, Critique x5 w/ CTRL reaches 16.24% Pass@1 vs 7.88% zero-shot (up to 106.1% relative gain).
- CTRL critic improves GPT-4o generator on CodeContests from 20.61% to 23.03% single-turn Pass@1 and 25.45% with 5 iterations.
- Single-turn CTRL critique lowers regression rate on CodeContests to 0.85% vs 3.03% for the SFT-only variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the critic with CTRL reinforcement learning rather than use zero-shot generation | Qwen2.5-Coder generator · CodeContests | Pass@1: 7.88 -> 11.76 | Helped |
| use CTRL instead of the Qwen2.5-Coder self-critic | Qwen2.5-Coder generator · CodeContests | Pass@1: 8.36 -> 11.76 | Helped |
| use CTRL instead of the GPT-4o critic | Qwen2.5-Coder generator · CodeContests | Pass@1: 10.67 -> 11.76 | Helped |
| use CTRL instead of the GPT-4o critic | Qwen2.5-Coder generator · LiveCodeBench (24.08-24.11) | Pass@1: 32.32 -> 33.21 | Helped |
| use CTRL instead of the GPT-4o critic | Qwen2.5-Coder generator · MBPP+ | Pass@1: 77.46 -> 78.84 | Helped |
| use CTRL instead of the Qwen2.5-Coder critic | GPT-4o generator · CodeContests | Pass@1: 20.24 -> 23.03 | Helped |
| use CTRL instead of the Qwen2.5-Coder critic | GPT-4o generator · LiveCodeBench (24.08-24.11) | Pass@1: 35.36 -> 33.39 | Hurt |
| use CTRL instead of the Qwen2.5-Coder critic | GPT-4o generator · MBPP+ | Pass@1: 76.67 -> 77.83 | Helped |
| perform five critique-revision rounds instead of one with CTRL (ablation) | Qwen2.5-Coder generator · CodeContests | Pass@1: 11.76 -> 16.24 | Helped |
| perform five critique-revision rounds instead of one with CTRL (ablation) | GPT-4o generator · CodeContests | Pass@1: 23.03 -> 25.45 | Helped |
| use SFT without the RL stage (ablation) | CTRL critic with Qwen2.5-Coder generator · CodeContests | Pass@1: 8.36 -> 11.76 | Hurt |
| use SFT without the RL stage (ablation) | CTRL critic · CodeContests | F1 Macro: 68.55 -> 69.10 | Hurt |
Surprising findings
- Self-critique with raw execution feedback is less effective than reasoning over execution feedback, and smaller critics (7B) often degrade performance in inference-only tests.
- CTRL-trained critic increases LiveCodeBench timeout rate (16.61% vs 10.54% zero-shot), a trade-off with accuracy.
Limitations
- PPO was unstable due to noisy value estimates, requiring switch to GRPO.
- Higher timeout rates on LiveCodeBench indicate efficiency trade-offs.
- Evaluation is limited to code generation plus JudgeBench; the critic is trained only on single-turn critiques.
Experimental setup
Datasets: TACO CodeContests LiveCodeBench MBPP+ JudgeBench
Benchmarks: CodeContests LiveCodeBench MBPP+ JudgeBench
Metrics: Pass@1 delta up (fix rate) delta down (regression rate) F1 accuracy
Baselines: Self-critique Critique w/ GPT-4o Critique w/ Qwen2.5-Coder Execution Feedback CTRL SFT-only variant Claude-3.5-Sonnet
Builds on: GRPO PPO Self-Refine Reflexion context distillation VeRL
Closest papers by meaning
- RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.90)
- Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning (ICLR 2026 · similarity 0.89)
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (ICLR 2024 · similarity 0.89)
- Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation (NeurIPS 2024 · similarity 0.88)
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.88)
- Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.