Research map / Large language models / Reasoning and chain of thought

CPMöbius: Iterative Coach–Player Reasoning for Data-Free Reinforcement Learning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CPMöbius, a cooperative Coach–Player RL loop where a Coach LLM proposes math tasks rewarded by the Player's validation-accuracy gain, and the Player trains with GRPO on majority-vote pseudo-labels, yielding data-free gains on math reasoning.

Problem

Reasoning LLMs rely on large human-curated tasks and labels for SFT or RL; the paper seeks data-free RL for mathematical reasoning without external training data during co-evolution.

Approach

Replace adversarial self-play with a cooperative Coach–Player loop: the Coach proposes difficulty-filtered tasks and is rewarded by the product of the Player's training reward and the validation accuracy gain (AMC), while the Player is trained with GRPO on majority-vote pseudo-labels.

Key results

• On Qwen2.5-Math-7B-Instruct, CPMöbius improves overall average by +4.9 (35.8 to 40.7) and OOD average by +5.4, vs RENT +1.5 overall and R-Zero +4.2 OOD.
• On Qwen2.5-Math-1.5B, overall average rises from 23.3 to 28.8, with Minerva going from 16.3 to 28.0.
• On OpenMath-Nemotron-1.5B, average improves from 59.5 to 62.1 where R-Zero failed to train.

Surprising findings

• R-Zero's method failed to train as challenger on OpenMath-Nemotron-1.5B.
• Under matched 60-step compute, CPMöbius slightly underperforms R-Zero on Qwen2.5-Math-1.5B (26.9 vs 27.1 average).

Limitations

• Uses AMC as validation signal for coach reward, which overlaps with the in-distribution benchmark (partially mitigated by OOD averages and alternative validation sets).
• Coach requires a warm-up on 4K external RL samples; without it performance drops.
• Only evaluated on mathematical reasoning.

Experimental setup

Datasets: AMC PRIME Eurus-2-RL-Data

Benchmarks: AMC AIME 2024 AIME 2025 Minerva MATH-500 OlympiadBench

Metrics: mean@k accuracy overall average OOD average

Baselines: R-Zero RENT Base Model

Builds on: GRPO RLVR R-Zero RENT self-consistency

Closest papers by meaning

• MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.88)
• BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.87)
• BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning (NeurIPS 2025 · similarity 0.86)
• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.86)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.86)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.86)