Research map / Reinforcement learning / Offline RL
RL's Razor: Why Online Reinforcement Learning Forgets Less
Idan Shenfeld, Jyothish Pari, Pulkit Agrawal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with RL (GRPO) instead of SFT at matched new-task performance | Qwen2.5-3B-Instruct, OpenVLA-7B · Math, Science Q&A, Tool use, robot pick-and-place | prior-task retention vs new-task accuracy | Improved |
| use forward KL on new task as predictor of forgetting (ablation) | ParityMNIST MLP · ParityMNIST/FashionMNIST | R^2 quadratic fit: 0.96 (forward KL) vs 0.34-0.93 for alternatives | Improved |
| forward KL as predictor in LLM experiments | Qwen2.5-3B-Instruct · LLM tasks | R^2: 0.96 (MNIST) -> 0.71 (LLM) | Mixed |
| train SFT on oracle minimum-KL distribution instead of arbitrary labels (ablation) | SFT on ParityMNIST · ParityMNIST/FashionMNIST | prior-task retention at matched accuracy | Improved |
| distill SFT from data generated by an RL-trained model (ablation) | SFT on ParityMNIST · ParityMNIST/FashionMNIST | accuracy-forgetting trade-off | Improved |
| use on-policy data without negative examples (1-0 Reinforce) instead of SFT (ablation) | Qwen2.5-3B-Instruct · Science Q&A | forgetting and KL shift | Improved |
| add negative examples to offline training (SimPO vs SFT) (ablation) | offline objective · Science Q&A | forgetting | No clear effect |
| use float32 instead of bfloat16 for RL training (ablation) | RL · weight update sparsity | update sparsity | No clear effect |
| increase model size from 3B to 14B under SFT (ablation) | Qwen2.5 SFT · Science Q&A | learning-forgetting trade-off | No clear effect |
| fine-tune with RL instead of SFT | Qwen2.5-3B-Instruct · Wikipedia representation similarity | CKNNA to base: 0.56 -> 0.94 | Improved |
Closest papers by meaning
- Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem (ICML 2024 · similarity 0.91)
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.90)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors (NeurIPS 2026 · similarity 0.89)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
