Research map / Reinforcement learning / RL from feedback
Vanishing Gradients in Reinforcement Finetuning of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RFT (policy gradient, incl. PPO) gradients for an input vanish when its pretrained reward standard deviation is small, demonstrates prevalence and harm on GRUE and controlled settings, and finds an initial SFT phase (even on 1% of inputs) overcomes it.
Problem
Reinforcement finetuning of language models with policy gradient methods (e.g., PPO) can make very slow progress on some inputs even when their expected reward is suboptimal; the cause of this optimization failure is unclear.
Approach
Proves that the expected policy gradient for an individual input is bounded by a term proportional to the reward standard deviation under the model (to the 2/3 power), so small reward variance implies vanishing gradients regardless of expected reward; shows this extends to PPO clipped and KL-regularized variants and that an initial SFT phase, even on a small fraction of inputs, mitigates it.
Key results
- On NarrativeQA, RFT (default PPO) reaches train reward 0.101 versus 0.537 for SFT followed by RFT; learning-rate, temperature, and entropy heuristics fail to exceed the default RFT reward.
- SFT on as few as 1% of input samples with 40% of the optimization steps lets RFT reach 96% of the reward achieved after a full SFT phase on some datasets.
- In the theoretical linear softmax setting, RFT optimization time scales as Omega(1/std^2) versus O(ln(1/std)) for SFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add an initial SFT phase before RFT | RFT · NarrativeQA train reward | reward: 0.101 ± 0.009 -> 0.537 ± 0.005 | Helped |
| add an initial SFT phase before RFT | RFT · NarrativeQA test reward | reward: 0.116 ± 0.000 -> 0.544 ± 0.003 | Helped |
| increase the learning rate, across the tested settings | RFT · NarrativeQA | reward: Train: 0.101 ± 0.009 -> 0.012 ± 0.010, 0.053 ± 0.010, 0.039 ± 0.018; test: 0.116 ± 0.000 -> 0.020 ± 0.017, 0.048 ± 0.016 | Hurt |
| apply temperature to the logits | RFT · NarrativeQA | reward: Train: 0.101 ± 0.009 -> 0.077 ± 0.021, 0.060 ± 0.018, 0.044 ± 0.004; test: 0.116 ± 0.000 -> 0.118 ± 0.002, 0.104 ± 0.009 | No clear effect |
| add entropy regularization | RFT · NarrativeQA | reward: Train: 0.101 ± 0.009 -> 0.080 ± 0.006, 0.024 ± 0.005, 0.011 ± 0.000; test: 0.116 ± 0.000 -> 0.113 ± 0.002, 0.019 ± 0.007 | Hurt |
| use 40% of SFT optimization steps and 1% of input samples before RFT (ablation) | RFT · NarrativeQA | reward relative to full-SFT reward: 96% | Helped |
| RFT rather than SFT finetuning | finetuning on NarrativeQA, ToTTo, and IMDB train samples · NarrativeQA | Pearson correlation: 0.05 -> 0.48 | Mixed |
| RFT rather than SFT finetuning | finetuning on NarrativeQA, ToTTo, and IMDB train samples · ToTTo | Pearson correlation: 0.16 -> 0.46 | Mixed |
| RFT rather than SFT finetuning | finetuning on NarrativeQA, ToTTo, and IMDB train samples · IMDB | Pearson correlation: 0.72 -> 0.72 | No clear effect |
Surprising findings
- Conventional heuristics (higher learning rate, temperature, entropy regularization) do not improve RFT over the default configuration on NarrativeQA.
- Adaptive batch gradient normalization (Adam) does not remedy vanishing gradients, since the per-input contribution remains negligible.
Limitations
- Experiments use small to moderate language models and do not include reward models learned from human feedback.
- Theoretical analysis restricted to linear softmax models with orthonormal inputs and gradient flow.
- Exploration of solutions is not exhaustive; only SFT and conventional heuristics are tested.
Experimental setup
Datasets: GRUE NarrativeQA ToTTo CommonGen IWSLT 2017 CNN/Daily Mail DailyDialog IMDB STS-B MNIST CIFAR10
Benchmarks: GRUE NarrativeQA ToTTo CommonGen STS-B MNIST CIFAR10
Metrics: mean reward reward standard deviation gradient norm Pearson correlation 10th percentile of pretrain reward std
Baselines: PPO RFT with default hyperparameters SFT increased learning rate temperature scaling entropy regularization
Builds on: PPO GRUE benchmark NLPO Adam
Closest papers by meaning
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.92)
- BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning (ICLR 2026 · similarity 0.90)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
- Learning Goal-Conditioned Representations for Language Reward Models (NeurIPS 2024 · similarity 0.89)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.89)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.