Research map / Large language models / Instruction and fine-tuning
Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, Jingren Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| inject RL-sampled tokens at high-divergence positions into base generations (forward cross-sampling) (ablation) | Qwen2.5-32B base with SimpleRL · AIME 2024 | Mean@32 accuracy (%): 8.23 -> >25 with 3.86% tokens | Improved |
| inject RL-sampled tokens into base generations (ablation) | Qwen2.5-32B base with SimpleRL · AIME 2025 | Mean@32 accuracy (%): 5.3 -> >14 with 1.53% tokens | Improved |
| inject DAPO tokens into base generations (ablation) | Qwen2.5-32B base with DAPO · AIME 2024 | Mean@32 accuracy (%): 8.23 -> >44 with 7.8% tokens | Improved |
| revert RL tokens to base samples at high-divergence positions (reverse cross-sampling) (ablation) | Qwen2.5-32B SimpleRL · AIME 2024 | Mean@32 accuracy (%): 25.52 -> <8.3 with 5% tokens | Hurt |
| revert RL tokens to base samples (ablation) | Qwen2.5-32B DAPO · AIME 2025 | Mean@32 accuracy (%): 32 -> <4.5 with 9.89% tokens | Hurt |
| use clip-higher (0.28) instead of 0.2 upper clip in DAPO (ablation) | Qwen2.5-Math-7B DAPO · token JS divergence | share near-zero divergence | Mixed |
| use SFT instead of RLVR (DAPO) | Qwen2.5-32B · token JS divergence | high-divergence set size: sigma 0.00143 (RL) vs 0.00347 (SFT) | Hurt |
| weight advantages up for high-KL tokens (sigmoid weighting) (ablation) | DAPO on Qwen2.5-Math-7B · AIME24/AIME25/AMC avg | accuracy (%): 42.48 -> 45.05 | Improved |
| weight advantages up for low-KL tokens (ablation) | DAPO on Qwen2.5-Math-7B · AIME24/AIME25/AMC avg | accuracy (%): 42.48 -> 44.92 | Improved |
| weight advantages up for high-KL tokens (ablation) | DAPO on Qwen2.5-Math-7B · AMC | accuracy (%): 75.08 -> 78.40 | Improved |
Closest papers by meaning
- On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.92)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.91)
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (NeurIPS 2026 · similarity 0.91)
- Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
