Research map / Reinforcement learning / RL from feedback
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RL finetuning of LLMs updates only a sparse 5-30% subnetwork of parameters (with full-rank updates), that finetuning this subnetwork reproduces the full model, and that in-distribution training drives the sparsity.
Problem
Whether RL finetuning of large language models actually updates all parameters, or only a small subnetwork, and what drives this update sparsity.
Approach
Measures bf16-level parameter update sparsity in publicly released RL checkpoints across many algorithms and models, shows that finetuning only the RL-identified subnetwork (via gradient masking) recovers full-finetuned performance and nearly identical weights, and attributes sparsity mainly to training on near-policy (in-distribution) data rather than KL or gradient clipping.
Key results
- Across published RL checkpoints, 68.5%-96.0% of parameters are unchanged after RL, while SFT updates are dense (6%-15% sparsity).
- Finetuning only the subnetwork matches or beats full finetuning: DPO average 41.9 vs 40.3 and PRIME overall 72.2 vs 69.8 on MATH500.
- Subnetwork overlap across seeds is about 60.5% versus 36.7% for random guessing; removing KL regularization changes GRPO sparsity only from 69.8% to 68.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune only the RL-identified subnetwork instead of all parameters (ablation) | DPO on Llama-3.1-Tulu-3-8B-SFT · DPO average | score: 40.3 -> 41.9; +1.6 | Helped |
| finetune only the RL-identified subnetwork instead of all parameters (ablation) | PRIME on Qwen2.5-Math-7B · MATH500 overall | score: 69.8 -> 72.2; +2.4 | Helped |
| add KL regularization (ablation) | GRPO on Qwen-2.5-7B-Instruct · GRPO training | update sparsity (%): 68.8% -> 69.8% | No clear effect |
| add gradient clipping (ablation) | GRPO on Qwen-2.5-7B-Instruct · GRPO training | update sparsity (%) | No clear effect |
| train on in-distribution rejection-sampled data rather than out-of-distribution data (ablation) | SFT and DPO · Qwen2.5-Math-7B RFT and Llama-3.1-8B-SFT DPO | update sparsity (%): 6.8 -> 91.2 | Helped |
| add SFT before RL (ablation) | RL finetuning · RL update sparsity | update sparsity | No clear effect |
| increase training duration (ablation) | PRIME · training checkpoints | update sparsity | Hurt |
| train DPO on out-of-distribution data without prior SFT (ablation) | DPO · Llama-3.1-8B-SFT and Zephyr-7b-Beta | update sparsity (%): 81.4 -> 6.8; 7.7 | Hurt |
| vary random seed (ablation) | Tulu-3-8B-SFT subnetworks · Subnetwork overlap | overlap (%): 36.7 -> 60.5; 36.7 -> 60.6 | Helped |
| vary training data (ablation) | Tulu-3-8B-SFT subnetworks · Subnetwork overlap | overlap (%): 14.6 -> 26.7; 36.7 -> 67.1 | Helped |
| vary seed, data, and RL algorithm (ablation) | DPO and PRIME subnetworks · Subnetwork overlap | overlap (%): 23.0 -> 59.1; 12.9 -> 33.2 | Helped |
Surprising findings
- Gradient clipping and KL regularization have limited effect on update sparsity.
- SFT on in-distribution rejection-sampled data yields about 90% sparsity, suggesting in-distribution data drives sparsity in SFT too.
Limitations
- Controlled experiments are computationally prohibitive, so many analyses rely on public checkpoints with unknown training configs.
- Only one factor varied at a time; interactions among factors are not studied.
- Subnetwork overlap across seeds, data and algorithms is well short of 100%.
Experimental setup
Datasets: allenai/llama-3.1-tulu-3-8b-preference-mixture PRIME-RL/Eurus-2-Rollout GSM8K MATH
Benchmarks: AGIEval LSAT-AR AGIEval LSAT-LR AGIEval LogiQA-en GPQA MMLU Pro Math MATH500
Metrics: accuracy update sparsity subnetwork overlap update rank
Baselines: full finetuning SFT LoRA random subnetwork
Builds on: Lottery Ticket Hypothesis DPO PPO GRPO PRIME
Closest papers by meaning
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.91)
- Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning (ICML 2025 · similarity 0.90)
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.89)
- Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models (ICLR 2025 · similarity 0.89)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.89)
- Behavior Injection: Preparing Language Models for Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.