Research map / Reinforcement learning / RL from feedback
Strategyproof Reinforcement Learning from Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows existing RLHF methods (Pessimistic Social Welfare, MaxMin-RLHF) are manipulable and that any strategyproof RLHF has k-times worse social welfare in the worst case, then proposes Pessimistic Median of MLEs, which is approximately strategyproof with suboptimality decaying with labelers and samples.
Problem
Offline RLHF from multiple strategic labelers who may misreport preferences to steer the learned policy toward their own reward functions, with the goal of maximizing average social welfare.
Approach
Combines per-labeler pessimistic confidence sets around Bradley-Terry MLEs with a coordinate-wise median over the sets, so that no single labeler can benefit much from misreporting; this trades off incentive alignment against social welfare, with incentive strength tied to uniform policy coverage.
Key results
- Pessimistic Social Welfare can be made to have social welfare near zero versus optimal BL when a single labeler misreports, giving suboptimality of at least BL-3 epsilon.
- Any strategyproof RLHF algorithm has suboptimality at least (k-1)/k, i.e., approximation ratio at most 1/k.
- Pessimistic Median of MLEs is O~(kappa_i sqrt(d/n))-strategyproof and has suboptimality O~(sqrt(d/k) + max_i kappa_i* k sqrt(d/n)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Pessimistic Median of MLEs rather than existing pluralistic RLHF methods | RLHF with strategic labelers · Contextual bandits | strategyproofness | Helped |
| Allow even one labeler to report strategically | Pessimistic Social Welfare · Contextual bandits | suboptimality: ≥ BL-3ε | Hurt |
| Require strategyproofness | Any RLHF algorithm · RLHF with k labelers | worst-case suboptimality: ≥ (k-1)/k | Hurt |
| Use Pessimistic Median of MLEs | RLHF with strategic labelers · Contextual bandits | strategyproofness: \tilde \mathcal{O}(κ_i √(d/n)) | Helped |
| Increase the number of labelers | Pessimistic Median of MLEs · Contextual bandits | suboptimality: √(d log(k/δ)/k) | Helped |
| Increase the number of samples | Pessimistic Median of MLEs · Contextual bandits | suboptimality: k√((d+log(k/δ))/n) | Helped |
| Have all labelers share the same reward function | Pessimistic Median of MLEs · Contextual bandits | suboptimality: const · max_{i ∈ [k]} κ_i* · k√((d+log(k/δ))/n) | Helped |
| Use Pessimistic Median of MLEs in an MDP | RLHF with strategic labelers · MDPs | strategyproofness: \tilde \mathcal{O}(ν_i √(d/n)) | Helped |
Surprising findings
- Any strategyproof RLHF method must incur constant suboptimality regardless of sample size, a fundamental trade-off between incentive and policy alignment.
- Approximate strategyproofness of the proposed method requires uniform policy coverage over all policies, not just coverage of the optimal policy.
Limitations
- Results hold under linear reward realizability, a hyperrectangle assumption on achievable policy features, and the Bradley-Terry preference model.
- Strategyproofness is approximate (epsilon-strategyproof) rather than exact, and the bounds depend on uniform coverage coefficients.
- No empirical evaluation; not validated on LLM fine-tuning.
Experimental setup
Metrics: suboptimality approximation ratio strategyproofness gap
Baselines: Pessimistic Social Welfare MaxMin-RLHF
Builds on: Pessimistic Social Welfare MaxMin-RLHF Gibbard-Satterthwaite theorem Bradley-Terry model
Closest papers by meaning
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.90)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.89)
- Reinforcement Learning from Imperfect Corrective Actions and Proxy Rewards (ICLR 2025 · similarity 0.89)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.89)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.