AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game

Barna Pásztor, Thomas Kleine Buening, Andreas Krause

ICLR 2026 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add the trained SLHF Follower refinement turn to the SLHF Leader responseSLHF · HelpSteer2 vs base modelpreference score: 0.734 -> 0.800Improved
add the Follower refinement over the Leader's own responseSLHF · HelpSteer2 self-refinementpreference score: 60.5%Improved
use SLHF-trained Follower instead of prompting the base model to refine (ablation)base model refining base responses · HelpSteer2 refinementpreference score: 0.549 -> 0.803Improved
use SLHF Follower instead of RLOO as Follower on RLOO Leader (ablation)RLOO Leader · HelpSteer2 refinementpreference score: 0.403 -> 0.665Improved
use SLHF Follower instead of Nash baseline as Follower on Nash Leader (ablation)Nash baseline Leader · HelpSteer2 refinementpreference score: 0.493 -> 0.600Improved
SLHF Leader versus Nash baseline first responseSLHF Leader · HelpSteer2preference score: 0.503No clear effect
increase test-time samples N from 1 to 5 for RLOORLOO · HelpSteer2 Best-of-Naverage preference vs base: 0.594 -> 0.338Hurt
increase test-time samples N from 1 to 5 for Nash baselineNash baseline · HelpSteer2 Best-of-Naverage preference vs base: 0.726 -> 0.718No clear effect
increase test-time samples N from 1 to 5 for SLHF (sequential refinement)SLHF · HelpSteer2 Best-of-Naverage preference vs base: 0.738 -> 0.765Improved
increase N from 1 to 5 on correctness for SLHFSLHF · HelpSteer2 Best-of-Ncorrectness preference: 0.670 -> 0.803Improved
increase follower weight kappa from 1 to 5 (ablation)SLHF Follower · HelpSteer2 validation (Qwen2.5-0.5B-Instruct)preference over base: 0.784 -> 0.806Improved
increase follower weight kappa from 1 to 5 (ablation)SLHF Leader · HelpSteer2 validation (Qwen2.5-0.5B-Instruct)preference over base: 0.761 -> 0.723Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy