Research map / Large language models / Reasoning and chain of thought

Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MT-GRPO, a multi-task GRPO variant that adapts task sampling weights via improvement-aware softmax reweighting and uses a ratio-preserving, acceptance-aware sampler so the weights match realized gradient contributions, improving worst-task accuracy on Countdown/Zebra/ARC and other reasoning mixtures.

Problem

Naively applying GRPO to multiple reasoning tasks yields imbalanced outcomes where some tasks dominate and others stagnate, and tasks differ in zero-advantage (zero-gradient) rates, distorting effective task contributions.

Approach

Decouple task reweighting from the GRPO loss: update softmax task-weight logits using task-level rewards plus per-step task improvements of the GRPO loss (λ-weighted), and use a ratio-preserving sampler with acceptance-aware oversampling so post-filtering batch proportions match the target weights.

Key results

• In the 3-task setting, MT-GRPO achieves 16–28% absolute worst-task accuracy improvement over GRPO and DAPO's 6% gain, with competitive average accuracy (per abstract).
• On Qwen2.5-7B with MATH+ARC+SciKnowEval, MT-GRPO (λ=0.2) reaches 37.6% worst-task accuracy vs 25.0% for DAPO and 15.0% for GRPO.
• MT-GRPO reaches DAPO's final worst-task accuracy of 25.0% in about 5.5 hours vs 13.3 hours for DAPO on the 7B setting.

Surprising findings

• Sequential per-family training, even with its strongest ordering, underperforms joint MT-GRPO on worst-task accuracy (46.3% vs 41.1%).
• Improvement-aware weighting alone (GRPO_IWU) can cut average accuracy substantially (52.0% to 43.9%) while raising worst-task accuracy, due to weight-to-gradient mismatch.

Limitations

• Robustness is only guaranteed within the training task mixture, not to out-of-mixture tasks.
• Relies on verifiable rewards; behavior in non-verifiable domains is uncharacterized.
• Trade-off parameter λ must be tuned and strongly shapes worst-task versus average accuracy.

Experimental setup

Datasets: Countdown Zebra puzzles ARC (string-based) SciKnowEval MATH

Benchmarks: Countdown Zebra puzzles ARC (string-based) SciKnowEval MATH

Metrics: worst-task accuracy average accuracy average per-task relative change training steps to reach worst-task threshold

Baselines: GRPO SEC-GRPO DAPO SEC-DAPO

Builds on: GRPO DAPO Self-evolving curriculum (SEC) distributionally robust optimization

Closest papers by meaning

• WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning (ICML 2026 · similarity 0.89)
• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.89)
• Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.88)
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale (NeurIPS 2025 · similarity 0.88)
• GPO: Learning from Critical Steps to Improve LLM Reasoning (NeurIPS 2025 · similarity 0.88)