Research map / Reinforcement learning / RL from feedback

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a multi-reward RLIF framework combining cluster-voting answer agreement and token-level self-certainty with per-channel GDPO normalization and KL-Cov regularization, trained on unlabeled MATH prompts with Qwen2.5-1.5B/3B.

Problem

Single internal-reward RLIF methods (self-certainty, majority voting) suffer reward hacking, entropy collapse and late-stage model collapse when trained without ground-truth supervision.

Approach

Decomposes the unsupervised reward into an answer-level cluster-voting reward and a completion-level self-certainty reward, normalizes each channel separately within the rollout group (GDPO), and adds a KL-Cov penalty only on the top 2% high-covariance tokens to prevent entropy collapse.

Key results

• On Qwen2.5-1.5B, multi-reward + KL-Cov reaches 51.6 on MATH500 and 29.9 on MMLU-Pro at step 160, vs INTUITOR's 22.6 and 24.5.
• At step 340 on Qwen2.5-1.5B, INTUITOR collapses to 0.0 GSM8K while multi-reward + KL-Cov retains 64.9 GSM8K (vs 73.2 for GRPO-GT at 340 steps).
• On Qwen2.5-3B, multi-reward + KL-Cov retains 95.2%/97.2%/99.5% of peak on GSM8K/MATH500/MMLU-Pro at step 340, close to GRPO with ground truth.

Surprising findings

• Cluster-voting-only training collapses via answer-space drift with rising entropy and lengthening completions, not via low-entropy templates.
• Naively summing rewards before normalization silences the lower-variance cluster channel by roughly an order of magnitude in logprob-space gradient norms.

Limitations

• Additional hyperparameters (beta_cov, alpha/beta weights) and only a symmetric alpha=beta=0.5 setting was explored.
• Relies on a sufficiently capable base model; no safeguards for poor initial reasoning.
• Multi-reward without KL-Cov still collapses around step 240-260 on 1.5B.

Experimental setup

Datasets: MATH-lighteval

Benchmarks: GSM8K MATH500 MMLU-Pro AIME 2024 AIME 2025 LiveCodeBench v6 CRUXEval-O

Metrics: pass@1 (greedy)

Baselines: INTUITOR GRPO with ground-truth reward Multi-reward (no KL-Cov)

Builds on: GRPO INTUITOR TTRL GDPO KL-Cov (entropy mechanism)

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• Learning to Reason without External Rewards (ICLR 2026 · similarity 0.90)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.89)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.89)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.89)