Research map / Large language models / Reasoning and chain of thought

Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Critique Reinforcement Learning (CRL), rewarding a model for correctly judging whether a (question, solution) pair is correct, and trains Critique-Coder by mixing 20% CRL with 80% standard RL data on Qwen3-4B/8B.

Problem

Standard RLVR for code reasoning models rarely elicits explicit critique or reflection on existing solutions, and prior critique-teaching methods (CFT, CGD) rely on teacher-provided critiques.

Approach

Add a critique task to RL: the policy judges a (question, solution) pair as True/False and receives a binary reward for matching the test-derived label, combined with the usual test-pass-rate RL reward in a hybrid GRPO setup with 20% CRL data.

Key results

• Critique-Coder-4B reaches 59.0 on LiveCodeBench v5 vs 54.2 for Qwen3-4B baseline and 56.6 for Qwen3-4B-RL.
• Critique-Coder-8B reaches 60.8 on LiveCodeBench v5 and 35.6 on Aider-Polyglot (+7.2 over Qwen3-8B baseline).
• On BBEH subtasks, Critique-Coder-4B averages 52.8 vs 46.7 for the Qwen3-4B baseline (+6.1).

Surprising findings

• Training with 100% CRL data drops average score to 44.5, below the RL-only baseline of 45.7.
• Critique-based parallel test-time scaling on post-hoc self-critique did not improve performance.

Limitations

• Self-critique on final answers without reasoning traces does not improve results.
• Weaker performance on non-Python languages (Aider-Polyglot lags GPT-o1), attributed to Python-only training.
• CRL-only training degrades performance, so CRL must be mixed with RL.

Experimental setup

Datasets: rStar-Coder seed dataset (filtered)

Benchmarks: EvalPlus BigCodeBench-Instruct Aider-Polyglot LiveCodeBench v5 BBEH

Metrics: pass@1 accuracy average score

Baselines: Qwen3-4B (Thinking) Qwen3-8B (Thinking) Qwen3-4B-RL Qwen3-8B-RL DeepSeek-R1-Distill-14B DeepCoder-14B DeepSeek-V2.5 GPT-o1 AceCoder-7B

Builds on: GRPO Critique Fine-Tuning rStar-Coder DeepCoder

Closest papers by meaning

• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.92)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (ICLR 2026 · similarity 0.89)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.89)