Research map / Reinforcement learning / RL from feedback
Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignment
Byeonghu Na, Hyungho Na, Yeongmin Kim, Suhyeon Jo, HeeSun Bae, Mina Kang, Il-chul Moon
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace reverse KL with Wasserstein policy regularization | PPO · TL;DR | win rate vs SFT: 0.848 -> 0.924 | Improved |
| replace reverse KL with Wasserstein policy regularization | PPO · HH-RLHF | win rate vs SFT: 0.828 -> 0.852 | Improved |
| replace reverse KL with Wasserstein policy regularization | PPO · MT-Bench | score: 4.000 -> 4.272 | Improved |
| replace reverse KL with Wasserstein policy regularization | CodeGemma-7B · APPS (all) | pass@1: 8.32 -> 8.79 | Improved |
| use Wasserstein penalty instead of forward KL | PPO · TL;DR | win rate vs RKL: 0.040 -> 0.608 | Improved |
| use Wasserstein penalty instead of chi-squared | PPO · TL;DR | win rate vs RKL: 0.540 -> 0.608 | Improved |
| scale policy to Gemma-7B with Gemma-7B embedding for cost matrix | Wasserstein PPO · TL;DR | win rate vs RKL-2B: 0.608 -> 0.712 | Improved |
| use Gemma-7B embeddings instead of Gemma-2B for cost matrix (2B policy) (ablation) | Wasserstein PPO · TL;DR | win rate vs SFT: 0.924 -> 0.908 | No clear effect |
| reduce Sinkhorn iterations from 10 to 5 (ablation) | Wasserstein PPO · TL;DR | win rate | Hurt |
| increase Sinkhorn iterations to 30 (ablation) | Wasserstein PPO · TL;DR | win rate | No clear effect |
| decrease truncation parameters k1,k2 or entropy strength lambda (ablation) | Wasserstein PPO · TL;DR | win rate | Hurt |
| change cost function from Euclidean to cosine distance (ablation) | Wasserstein PPO · TL;DR | win rate | No clear effect |
Closest papers by meaning
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.90)
- KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity (NeurIPS 2025 · similarity 0.90)
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.90)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.89)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.89)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
