Research map / Reinforcement learning / RL from feedback
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a baseline to KL-DPG (GDC++ vs GDC) (ablation) | GDC with GPT-2 small · 10 controlled generation tasks | constraint satisfaction, KL to p and to a | Helped |
| add a baseline to KL-DPG (ablation) | GDC · tasks 1-10 | training stability | Helped |
| GDC++ with batch size 256 instead of 2048 | GDC++ · task 1 (word amazing) | samples to reach E[phi]=0.5: 4M -> 1M | Helped |
| remove baseline (GDC) at batch size 256 (ablation) | GDC++ · task 1 (word amazing) | constraint satisfaction E[phi] | Hurt |
| add baseline, small batch sizes (256 to 2048 tested) (ablation) | GDC · tasks 1 and 8 | constraint satisfaction and KL to p | Helped |
| add baseline to DPG (ablation) | GDC · tasks 1 and 7, 3 seeds | gradient variance Var(G) | Helped |
| add baseline to DPG (ablation) | GDC · tasks 1 and 7, 3 seeds | advantage variance Var(A) | Helped |
| GDC/GDC++ distribution matching instead of Reinforce reward maximization | GPT-2 small · tasks 1-6 | Self-BLEU-5, Distinct-1 diversity | Helped |
| GDC/GDC++ instead of Ziegler KL-penalty reward objective | GPT-2 small · tasks 1-6 | deviation from a and training stability | Helped |
| Reinforce instead of GDC for constraint satisfaction speed | GPT-2 small · tasks 1-6 | constraint satisfaction speed | Hurt |
Closest papers by meaning
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.90)
- Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data (ICML 2024 · similarity 0.90)
- BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback (ICML 2024 · similarity 0.90)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.90)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.89)
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.