Research map / Large language models / Reasoning and chain of thought

Advancing LLM Reasoning with Natural Language and Numerical Feedback

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Critique-GRPO, an online RL framework that trains LLMs on both initial responses and critique-guided self-refinements, using natural language critiques alongside scalar rewards.

Problem

Numerical-only RL for LLM reasoning plateaus, yields ineffective spontaneous self-reflection, and leaves persistent failures on some training problems; natural language feedback has mainly been used offline via SFT.

Approach

Integrate natural language critiques (indicative or CoT, from rule-based or model-based reward systems) into the online GRPO loop: sample initial responses, refine failed groups in-context using critiques, and train on both sets with a shared baseline and a policy-shaping ratio that upweights low-probability refinement tokens.

Key results

• Critique-GRPO (CoT critique) reaches 47.08 average Pass@1 on Qwen2.5-7B-Base vs 41.18 for R1-GRPO across eight tasks.
• On Qwen3-8B, Critique-GRPO with CoT critique averages 68.26 vs 63.75 for R1-GRPO, and self-critique reaches 60.00 Avg@32 on AIME24 vs 53.33 for R1-GRPO.
• With only 4k prompts, Critique-GRPO on Qwen2.5-Math-7B-Base averages 51.06 vs 46-47 range for numerical-feedback baselines trained on 46k prompts.

Surprising findings

• Inverting critique conclusions (adversarial critiques) causes entropy explosion and performance degradation.
• Longer, more detailed CoT critiques (~2079 tokens) slightly reduce performance compared with ~683-token critiques.

Limitations

• Refinement is triggered only when all initial responses in a group fail, adding overhead and depending on critique quality.
• Reliance on a strong critique model (GPT-4o or 405B) in some settings; critique quality determines gains.
• Evaluation uses greedy decoding with pass@1 over three runs, and some benchmarks (e.g., AIME) have very small sample sizes.

Experimental setup

Datasets: OpenR1-Math-220k NuminaMath 1.5

Benchmarks: MATH-500 Minerva-Math OlympiadBench AMC23 AIME24 AIME25 TheoremQA GPQA-Diamond MMLU-Pro

Metrics: Pass@1 Avg@32 pass@k

Baselines: SFT RAFT Refinement FT Critique FT CITL-FT R1-GRPO R1-Dr.GRPO SimpleRL-Zero PRIME-Zero Oat-Zero

Builds on: GRPO Dr. GRPO RAFT VERL Critique Fine-Tuning

Closest papers by meaning

• Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning (ICLR 2026 · similarity 0.92)
• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning (NeurIPS 2026 · similarity 0.92)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.92)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.92)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.92)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.92)