Research map / Reinforcement learning / RL from feedback
Critique-RL: Training Critiquing Language Models Through Two-Stage RL for Improved Discrimination and Constructive Feedback
Zhiheng Xi, Jixuan Huang, Xin Guo, Boyang Hong, Dingwen Yang, Xiaoran Fan, Shuo Li, Zehui Chen, Junjie Ye, Siyu Yuan, Zhengyin Du, Xuesong Yao and 6 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Stage I discrimination training (ablation) | Critique-RL · MATH (Qwen2.5-3B) | Acc@Refine: 45.9 -> 48.6 | Improved |
| add Stage II helpfulness training (ablation) | Critique-RL · MATH (Qwen2.5-3B) | Acc@Dis: 78.7 -> 82.8 | Improved |
| add Stage I (ablation) | Critique-RL · AQuA (Qwen2.5-3B) | Acc@Refine: 53.9 -> 56.7 | Improved |
| keep discrimination reward and KL regularization in Stage II (ablation) | Critique-RL · MATH (Qwen2.5-3B) | Acc@Dis: 77.7 -> 82.8 | Improved |
| keep discrimination reward and KL in Stage II (ablation) | Critique-RL · AQuA (Qwen2.5-3B) | Acc@Dis: 61.6 -> 69.9 | Improved |
| use rrefine reward instead of r_delta in Stage II (ablation) | Critique-RL · MATH (Qwen2.5-3B) | Acc@Refine: 48.2 -> 48.6 | Improved |
| use rrefine reward instead of rcorrection in Stage II (ablation) | Critique-RL · MATH (Qwen2.5-3B) | Acc@Refine: 47.7 -> 48.6 | Improved |
| add a second training iteration (ablation) | Critique-RL · MATH (Qwen2.5-3B) | Acc: 48.6 -> 51.0 | Improved |
| use online RL critic instead of SFT critic | Critique-RL vs SFT · AQuA (Qwen2.5-3B) | Acc: 46.46 -> 56.69 | Improved |
| use Critique-RL critic instead of Directly RL actor | Qwen2.5-7B · MATH | Acc: 49.78 -> 58.40 | Improved |
| use Critique-RL instead of SCoRe | Qwen2.5-7B · MATH | Acc@Dis: 72.51 -> 85.20 | Improved |
| change beta from 0.01 to 0.012 (ablation) | Critique-RL · GSM8K (Qwen2.5-3B) | Acc: 75.89 -> 74.22 | No clear effect |
Closest papers by meaning
- Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning (ICLR 2026 · similarity 0.91)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.90)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.89)
- CriticEval: Evaluating Large-scale Language Model as Critic (NeurIPS 2024 · similarity 0.89)
- The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement (NeurIPS 2025 · similarity 0.89)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
