Research map / Reinforcement learning / RL from feedback
Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game
Barna Pásztor, Thomas Kleine Buening, Andreas Krause
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the trained SLHF Follower refinement turn to the SLHF Leader response | SLHF · HelpSteer2 vs base model | preference score: 0.734 -> 0.800 | Improved |
| add the Follower refinement over the Leader's own response | SLHF · HelpSteer2 self-refinement | preference score: 60.5% | Improved |
| use SLHF-trained Follower instead of prompting the base model to refine (ablation) | base model refining base responses · HelpSteer2 refinement | preference score: 0.549 -> 0.803 | Improved |
| use SLHF Follower instead of RLOO as Follower on RLOO Leader (ablation) | RLOO Leader · HelpSteer2 refinement | preference score: 0.403 -> 0.665 | Improved |
| use SLHF Follower instead of Nash baseline as Follower on Nash Leader (ablation) | Nash baseline Leader · HelpSteer2 refinement | preference score: 0.493 -> 0.600 | Improved |
| SLHF Leader versus Nash baseline first response | SLHF Leader · HelpSteer2 | preference score: 0.503 | No clear effect |
| increase test-time samples N from 1 to 5 for RLOO | RLOO · HelpSteer2 Best-of-N | average preference vs base: 0.594 -> 0.338 | Hurt |
| increase test-time samples N from 1 to 5 for Nash baseline | Nash baseline · HelpSteer2 Best-of-N | average preference vs base: 0.726 -> 0.718 | No clear effect |
| increase test-time samples N from 1 to 5 for SLHF (sequential refinement) | SLHF · HelpSteer2 Best-of-N | average preference vs base: 0.738 -> 0.765 | Improved |
| increase N from 1 to 5 on correctness for SLHF | SLHF · HelpSteer2 Best-of-N | correctness preference: 0.670 -> 0.803 | Improved |
| increase follower weight kappa from 1 to 5 (ablation) | SLHF Follower · HelpSteer2 validation (Qwen2.5-0.5B-Instruct) | preference over base: 0.784 -> 0.806 | Improved |
| increase follower weight kappa from 1 to 5 (ablation) | SLHF Leader · HelpSteer2 validation (Qwen2.5-0.5B-Instruct) | preference over base: 0.761 -> 0.723 | Hurt |
Closest papers by meaning
- Nash Learning from Human Feedback (ICML 2024 · similarity 0.93)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.92)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.92)
- Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.91)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
