Research map / Reinforcement learning / RL from feedback
KL-Regularized Reinforcement Learning is Designed to Mode Collapse
Anthony GX-Chen, Jatin Prakash, Jeff Guo, Rob Fergus, Rajesh Ranganath
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MARA reward augmentation (reverse KL) to RLOO-style training | RLOO · Creative QA, out-of-dist reward | Out-dist. Reward: 1.280 -> 1.451 | Improved |
| use forward KL with MARA instead of reverse KL (ablation) | MARA · Creative QA, Mean Distinct | Mean Distinct: 4.14 -> 4.62 | Improved |
| add MARA to RLOO | RLOO · Creative QA, Semantic Div | Semantic Div: 0.192 -> 0.186 | Hurt |
| add MARA reward augmentation to REINVENT | REINVENT · SYNTH, threshold 0.85 | OB100: 4114 -> 3654 | Improved |
| add MARA reward augmentation to REINVENT | REINVENT · ALL-AMIDE, threshold 0.85 | Yield: 1098 -> 1235 | Improved |
| add MARA reward augmentation to REINVENT | REINVENT · SYNTH, threshold 0.80 | Circles: 67 -> 59 | Hurt |
| add MARA reward augmentation to REINVENT | REINVENT · SYNTH/ALL-AMIDE | IntDiv1: 0.766 -> 0.761 | No clear effect |
| add MARA mode-anchored reward augmentation | Qwen2.5-3B with naive KL regularization · Verifiable 1-2 task | answer diversity (entropy) | Improved |
| add MARA to reverse and forward KL | Qwen2.5-3B · Verifiable 1-2 task | correctness vs diversity Pareto front | Improved |
| lower KL regularization strength beta (ablation) | KL-regularized RL · didactic two-mode example | mode coverage | Hurt |
| lower KL regularization strength beta with equal rewards (ablation) | KL-regularized RL · didactic equal-reward example | probability of low-support correct mode | No clear effect |
| switch from reverse KL to forward KL regularization (ablation) | KL-regularized RL · didactic two-mode example | mode coverage | No clear effect |
Closest papers by meaning
- Sharp Analysis for KL-Regularized Contextual Bandits and RLHF (NeurIPS 2025 · similarity 0.90)
- Logarithmic Regret for Online KL-Regularized Reinforcement Learning (ICML 2025 · similarity 0.90)
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.89)
- Demonstration-Regularized RL (ICLR 2024 · similarity 0.89)
- Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability (NeurIPS 2026 · similarity 0.89)
- Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
