Research map / Large language models / Reasoning and chain of thought
RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tango jointly trains a Qwen2.5-Math-7B generator and a generative process-level Qwen2.5-7B verifier with interleaved RL, using only outcome-level verification rewards, reaching SOTA among 7B/8B models on competition math, out-of-domain reasoning, and ProcessBench.
Problem
Existing RL post-training for LLM reasoning uses fixed or SFT-trained verifiers (reward models), which are prone to reward hacking and generalize poorly; co-training generator and verifier has been underexplored.
Approach
Train both the generator and a generative step-level verifier with RL in an interleaved loop. The verifier is rewarded only for matching the outcome correctness of the generator's answer, so step-level judgments emerge without process labels. Step rewards are blended with outcome advantages using a decaying alpha, and the verifier uses class-aware advantage reweighting to avoid collapse.
Key results
- GRPO w/ Tango reaches 81.4 on MATH500 and 20.0 on AIME2025, versus 74.6 and 10.0 for vanilla GRPO (7B).
- Tango-Qwen-7B achieves 49.5 average on math and 62.8 average on out-of-domain tasks, state-of-the-art among 7B/8B models in the paper's comparison.
- Tango verifier reaches 43.9 average F1 on ProcessBench, above Skywork-PRM-7B (42.1), without any process labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Tango co-training to GRPO | Qwen2.5-Math-7B generator · AIME 2025 | pass@1 accuracy: 10.0 -> 20.0 | Helped |
| Add Tango co-training to GRPO | Qwen2.5-Math-7B generator · Mathematical reasoning average | pass@1 accuracy: 37.0 -> 46.1 | Helped |
| Add Tango co-training to GRPO | Qwen2.5-Math-7B generator · Out-of-domain reasoning average | pass@1 accuracy: 57.6 -> 61.1 | Helped |
| Train the generator and verifier jointly instead of fixing either component (ablation) | Tango on the last-letter concatenation task · Algorithmic reasoning test data | generator accuracy | Helped |
| Fix the generator during verifier training (ablation) | Tango on the last-letter concatenation task · Algorithmic reasoning test data | verifier F1 | Hurt |
| Fix the verifier during generator training (ablation) | Tango on the last-letter concatenation task · Algorithmic reasoning test data | generator accuracy | Hurt |
| Add an exponential decay schedule for the step-level reward weight (ablation) | Tango generator training · Mathematical reasoning | pass@1 accuracy | Helped |
| Normalize step-level and outcome-level advantages separately before combining them (ablation) | Tango generator training · Mathematical reasoning | learning stability | Helped |
| Remove class-aware reweighting from verifier advantage computation (ablation) | Tango verifier · Verifier training with early class imbalance | verification performance | Hurt |
| Use Tango’s RL-trained generative verifier instead of an outcome-only reward | GRPO generator · Mathematical and out-of-domain reasoning benchmarks | pass@1 accuracy | Helped |
Surprising findings
- The verifier collapses to always predicting incorrect without class-aware advantage reweighting in early training.
- Qwen2.5-7B verifier starts weaker than the generator on math, yet mutual RL co-training still yields gains.
Limitations
- Verifier is initially weaker than the generator on math, requiring a 40-step warmup.
- Step-level reward blending requires careful alpha decay to avoid reward hacking.
- Step-level verification is not directly measurable on the math training data without gold step labels.
Experimental setup
Datasets: Eurus-2-SFT-Data Eurus-2-RL-Data ProcessBench AIME 2024 AIME 2025 AMC 2023 MATH-500 OlympiadBench BoardgameQA CRUXEval StrategyQA TableBench
Benchmarks: AIME 2025 AIME 2024 AMC 2023 MATH-500 OlympiadBench BoardgameQA CRUXEval StrategyQA TableBench ProcessBench
Metrics: pass@1 accuracy F1 score on ProcessBench
Baselines: GRPO RLOO REINFORCE++ PRIME Eurus-2-7B-PRIME Qwen2.5-Math-7B-Instruct rStar-Math-7B QwQ-32B-Preview Skywork-PRM-7B EurusPRM-7B
Builds on: GRPO RLOO REINFORCE++ PRIME Qwen2.5-Math
Closest papers by meaning
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.92)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)
- RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.91)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.91)
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.91)
- RAST: Reasoning Activation in LLMs via Small-model Transfer (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.