Research map / Reinforcement learning / RL from feedback
DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes GRPO under binary rewards, showing a question-level difficulty-bias weight, and proposes DisCO, a discriminative constrained optimization objective with non-clipping scoring functions, KL trust-region constraint via squared-hinge penalty, and a DRO partial-AUC term for imbalanced rollouts.
Problem
GRPO-style RL for large reasoning models has question-level difficulty bias from its group-relative advantage normalization, plus entropy instability (collapse or excessive growth) and imbalance between positive and negative rollouts.
Approach
Recast the RL objective as a discriminative objective that raises scores of positive (correct) outputs and lowers negative ones, with no per-question weight, using non-clipping log-likelihood or likelihood-ratio scores, a KL trust-region constraint enforced by squared-hinge penalty, and a KL-DRO (partial AUC) surrogate to handle imbalanced rollouts.
Key results
- For the 1.5B model, DisCO (log-L) with 8k train/test length reaches 0.533 average versus 0.457 for GRPO and 0.473 for DAPO, a ~7-point gain over GRPO.
- For the 7B model, DisCO (L-ratio) reaches 0.627 average versus 0.592 for GRPO.
- For the 8B Llama model, DisCO (log-L) reaches 0.584 average versus 0.516 for GRPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DisCO (log-L) instead of GRPO | DeepSeek-R1-Distill-Qwen-1.5B fine-tuning · Average across six benchmarks | pass@1 average: 0.457 -> 0.533 | Helped |
| Use DisCO (log-L) instead of DAPO | DeepSeek-R1-Distill-Qwen-1.5B fine-tuning · Average across six benchmarks | pass@1 average: 0.473 -> 0.533 | Helped |
| Use DisCO instead of GRPO | DeepSeek-R1-Distill-Qwen-7B fine-tuning · Average across six benchmarks | pass@1 average: 0.592 -> 0.627 | Helped |
| Use DisCO instead of GRPO | DeepSeek-R1-Distill-Llama-8B fine-tuning · Average across six benchmarks | pass@1 average: 0.516 -> 0.584 | Helped |
| Remove the hard-negative weighting by using DisCO rather than DisCO-b (ablation) | DisCO-b with L-ratio scoring · AIME datasets and other evaluated settings | performance | Helped |
| Use a non-clipping scoring function instead of clipping with ε_high = 0.2 (ablation) | DisCO-b on Qwen-1.5B · Training and generation entropy | performance and entropy stability | Helped |
| Use a non-clipping scoring function instead of clipping with ε_high = 0.28 (ablation) | DisCO-b on Qwen-1.5B · Training and generation entropy | performance and entropy stability | Helped |
| Use constrained optimization instead of KL regularization (ablation) | DisCO on Qwen-1.5B and Qwen-7B · Training | performance and training stability | Helped |
| Vary τ within {0.5, 1, 5, 10} (ablation) | DisCO on Qwen-1.5B · Training | performance | No clear effect |
| Add question-level weight bias √p(q)(1-p(q)) (ablation) | DisCO-b · Question accuracy during training | ratio of questions reaching correctness extremes | Hurt |
| Use clipped scoring with ε_high = 0.2 (ablation) | DisCO-b · Qwen-1.5B training | generation entropy | Hurt |
| Use clipped scoring with ε_high = 0.28 (ablation) | DisCO-b · Qwen-1.5B training | generation entropy | Hurt |
Surprising findings
- DisCO with 8k training and inference outperforms DeepScaleR-1.5B-Preview, which was trained with 24k length and evaluated with 32k length.
- Removing the question-level sqrt(p(1-p)) weight alone speeds up learning toward 100% correct and reduces 0% correct questions.
Limitations
- Restricted to binary rewards; non-binary rewards need future discriminative ranking extensions.
- Evaluated only on mathematical reasoning with distilled DeepSeek-R1 models up to 8B.
- Sensitive to hyperparameters delta, beta and tau were tuned per scoring function, and comparisons omit dynamic sampling.
Experimental setup
Datasets: DeepScaleR-Preview-Dataset DAPO-Math-17k
Benchmarks: AIME 2024 AIME 2025 MATH 500 AMC 2023 Minerva OlympiadBench
Metrics: pass@1 averaged over 16 responses
Baselines: GRPO GRPO-ER Dr. GRPO DAPO TRPA STILL-3-1.5B-preview DeepScaleR-1.5B-Preview GRPO-LEAD-7B
Builds on: GRPO DAPO Dr. GRPO TRPA TRPO
Closest papers by meaning
- CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models (NeurIPS 2025 · similarity 0.88)
- Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning (ICLR 2026 · similarity 0.88)
- WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning (ICML 2026 · similarity 0.88)
- All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.87)
- On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.