Research map / Reinforcement learning / RL from feedback

Degeneration-free Policy Optimization: RL Fine-Tuning for Language Models without Degeneration

ICML 2024 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Degeneration-free Policy Optimization (DfPO), an RL fine-tuning method for LMs that uses KL-masking and truncated advantages to improve task scores without KL-penalty hyperparameter search or text degeneration.

Problem

RL fine-tuning of language models with PPO and a KL-divergence penalty is highly sensitive to the penalty coefficient and often leads to text degeneration, where generated text becomes unnatural while task reward increases.

Approach

Reformulates policy gradient as separate likelihood maximization and minimization. KL-masking keeps only tokens whose likelihood can be moved in the advantage direction without leaving the reference policy (sign of log-ratio matches sign of advantage), and truncated advantages route each token to max or min, replacing the KL reward penalty with a hyperparameter-free rule.

Key results

• On IMDB text continuation with GPT-2, DfPO raises sentiment score while keeping initial perplexity, without KL hyperparameter search, and reaches a better sentiment-perplexity frontier than tuned PPO and DPO (Figure 3).
• On CommonGen with T5, DfPO+Supervised reaches METEOR 0.336 vs 0.329 for PPO+Sup and 0.326 for NLPO+Sup, with BERTScore and SPICE not degraded.
• PPO and DPO are highly sensitive to the KL coefficient, with some settings degrading perplexity sharply during training (Figure 1, Figure 4).

Measured effects reported in the paper

Surprising findings

• PPO with KL-regularization is highly sensitive to the penalty coefficient, so even a sufficient sweep yields the worst sentiment-perplexity frontier in the authors' experiments.

Limitations

• Assumes the reference policy is a sufficiently fluent language model.
• Evaluated only on small models (GPT-2, T5) and GPT-J-6B; not applied to RLHF/RLAIF with recent LLMs.
• On CommonGen, DfPO shows similar or slightly different diversity, with MSTTR being an exception.

Experimental setup

Datasets: IMDB RealToxicityPrompts CommonGen

Benchmarks: IMDB text continuation RealToxicityPrompts text detoxification CommonGen

Metrics: sentiment score perplexity non-toxicity score METEOR ROUGE BLEU BERTScore CIDEr SPICE diversity metrics

Baselines: PPO NLPO DPO Zero Shot Supervised

Builds on: PPO NLPO DPO RL4LMs GAE

Closest papers by meaning

• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.88)
• Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.88)
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.87)
• VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.87)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.87)
• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.87)