Research map / Large language models / Language, safety and interpretability
Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li, Jiahui Geng, Wenxi Li, Longyue Wang, Chenyang Lyu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| keep L18-20 MLP weights from RLVR (vs resetting Anchor to base) (ablation) | spurious-RLVR Qwen2.5-Math-7B · MATH-500 leakage subset | accuracy: 98% -> 86% when reset | Improved |
| keep L21-22 MLP weights from RLVR (vs resetting Adapter to base) (ablation) | spurious-RLVR Qwen2.5-Math-7B · leakage subsets MATH-500/MinervaMath | accuracy: drops of -6% and -10% when reset | Improved |
| keep only Anchor or only Adapter RLVR layers (ablation) | spurious-RLVR Qwen2.5-Math-7B · MinervaMath | accuracy: Keep Only Adapter 32%, Keep Only Anchor 42%, Keep Both 46% (base 50%) | Hurt |
| reset Anchor/Adapter layers (ablation) | spurious-RLVR Qwen2.5-Math-7B · LiveMathBench (leakage-free) | accuracy: stable at 70% | No clear effect |
| reset Anchor layers (ablation) | spurious-RLVR Qwen2.5-Math-7B · stable (non-leaked) samples | accuracy | No clear effect |
| reset Anchor layers (ablation) | spurious-RLVR Qwen2.5-Math-1.5B · MATH-500 leakage subset (38 cases) | accuracy: 39.5% -> 2.6% when reset | Improved |
| scale key activations of top-10 task neurons at Layer 18 (suppress/amplify) | spurious-RLVR Qwen2.5-Math-7B · leakage datasets | accuracy: -3.8% suppression, +4.4% amplification | Mixed |
| scale neuron keys at Layer 18 | spurious-RLVR Qwen2.5-Math-7B · leakage-free dataset | accuracy | No clear effect |
| scale neuron keys at Layer 25 (Structural Adapter) | spurious-RLVR Qwen2.5-Math-7B · leakage datasets | accuracy | Hurt |
| add probe-gated Layer-19 neuron suppression | RLVR-tuned Qwen2.5-Math-7B · MATH-500 wrong->right | accuracy: 100.0% -> 85.16% | Hurt |
| add probe-gated Layer-19 neuron suppression | RLVR-tuned Qwen2.5-Math-7B · AIME-2025 / LiveMathBench | accuracy: 30.0% -> 30.0%; 21.0% -> 21.0% | No clear effect |
Closest papers by meaning
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.90)
- Spurious Rewards: Rethinking Training Signals in RLVR (ICML 2026 · similarity 0.90)
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.89)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.89)
- Detecting RLVR Training Data via Structural Convergence of Reasoning (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
