Research map / Reinforcement learning / RL from feedback
Escaping Policy Contraction: Contraction-Aware PPO (CaPPO) for Stable Language Model Fine-Tuning
Dun Yuan, Di Wu, Xue Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace PPO with CaPPO (Pareto multi-objective updates plus entropy scheduling) | PPO · HH-RLHF, Qwen2.5-14B | win rate vs SFT (%): 65.1 ± 0.7 -> 69.0 ± 0.5 | Improved |
| replace PPO with CaPPO | PPO · UltraFeedback, Llama-3-8B-Instruct | win rate vs SFT (%): 62.4 ± 0.8 -> 66.9 ± 0.7 | Improved |
| add entropy scheduling to PPO (ablation) | PPO · HH-RLHF, Qwen2-7B | win rate vs SFT (%): 62.8 ± 0.7 -> 64.3 ± 0.7 | Improved |
| add entropy scheduling to PPO (ablation) | PPO · ShareGPT-1K, Llama3-8B | entropy (nats/token): 3.42 -> 3.76 | Improved |
| add entropy scheduling to PPO (ablation) | PPO · ShareGPT-1K, Llama3-8B | Repeat-2 (%): 24.5 -> 18.0 | Improved |
| add entropy scheduling to PPO (ablation) | PPO · ShareGPT-1K | SRR@tau: 0.37 -> 0.58 | Improved |
| use adaptive instead of fixed entropy coefficient in PPO (ablation) | PPO · HH-RLHF | SRR: 0.43 -> 0.59 | Improved |
| add Pareto multi-gradient update to adaptive-entropy PPO (ablation) | PPO (adaptive beta) · HH-RLHF | win rate (%): 65.1 -> 67.8 | Improved |
| use Pareto update instead of scalarized entropy weight (ablation) | PPO (lambda_H=0.3) · Summarize-from-Feedback | win rate (%): 58.0 -> 61.9 | Improved |
| raise scalarized entropy weight in PPO from 0.0 to 0.1 (ablation) | PPO · Summarize-from-Feedback | win rate (%): 57.2 -> 58.3 | Improved |
| replace PPO with CaPPO | PPO · RLHF macro-average, Qwen2-7B | SRR: 0.55 -> 0.82 | Improved |
| use off-policy methods (RRHF) instead of on-policy PPO | PPO · RLHF macro-average, Qwen2-7B | win rate vs SFT (%): 67.4 -> 64.7 win, SRR 0.55 -> 0.92 | Mixed |
Closest papers by meaning
- Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT (NeurIPS 2026 · similarity 0.91)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.90)
- Stabilizing Policy Optimization via Logits Convexity (NeurIPS 2026 · similarity 0.90)
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (ICML 2024 · similarity 0.89)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.89)
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
