Research map / Large language models / Reasoning and chain of thought

Probing RLVR Training Instability through the Lens of Objective-Level Hacking

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Frames RLVR training instability in MoE models as "objective-level hacking" caused by token-level credit misalignment, and uses controlled weight-injection experiments on a 30B MoE model to link this to growth of the training-inference discrepancy.

Problem

RLVR training for MoE LLMs is prone to instability, and the steady growth of training-inference discrepancy that accompanies it lacks a mechanistic explanation.

Approach

Treats token-level reweighting (clipping, TIS, injected weight distortion) and training-inference mismatch as biasing the GRPO objective by a spurious term Delta J = Cov(X, rho^{-1}); this "objective-level hacking" creates a positive feedback loop with discrepancy growth.

Key results

• Stronger token-level clipping (right range 0.2 to 0.28) correlates with faster growth of training-inference discrepancy, while sequence-level clipping (GSPO) shows no comparable anomalous growth.
• On the 30B MoE model, mismatch rises from 1.418e-2 to 2.664e-2 over training, versus 1.059e-2 to 1.081e-2 for the dense 7B model.
• Injecting a modest 20% upweighting of low-probability tokens into GSPO triggers discrepancy growth, while zero-mean Gaussian variance injection does not.

Surprising findings

• Stronger token-level clipping accelerates training-inference discrepancy growth, counter to its intended stabilizing role.
• Zero-mean variance-based weight noise does not induce discrepancy growth; only biased weight distortion does.

Limitations

• Theory relies on first-order and independence approximations; the paper acknowledges the derivation is approximate.
• Evidence is mainly from one 30B MoE model and one dense 7B model with a single benchmark (AIME24), and the authors note other factors may contribute.
• Mechanism for MoE-specific faster growth (expert activation inconsistency) is posited rather than established.

Experimental setup

Datasets: DAPO-Math-17k AIME24 Skywork-OR1-RL-Data

Benchmarks: AIME24

Metrics: validation accuracy training-inference discrepancy (std of importance weights) token entropy Pearson correlation coefficient

Baselines: GRPO with token-level clipping GRPO with TIS GSPO

Builds on: GRPO GSPO DAPO truncated importance sampling

Closest papers by meaning

• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.90)
• The Unlearnability Phenomenon in RLVR for Language Models (ICML 2026 · similarity 0.89)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.89)
• IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.89)