Research map / Reinforcement learning / RL from feedback
Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework
Kihyun Kim, Jiawei Zhang, Asuman E. Ozdaglar, Pablo A. Parrilo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase softmax beta in F^beta (ablation) | F^beta · MovieLens tabular | win rate vs uniform: 0.5987 -> 0.7784 | Improved |
| increase softmax beta in F^beta (ablation) | F^beta · MovieLens tabular | PPA level: 0.4869 -> 0 | Hurt |
| use F^beta instead of RLHF | RLHF · MovieLens tabular | PBM policy gain: 0.0611 -> 8.896e-4 (beta=1) | Improved |
| increase beta from 0 to 1 (ablation) | F^beta · Synthetic color | win rate: 0.6157 -> 0.8429 | Improved |
| increase beta from 0 to 1 (ablation) | F^beta · Synthetic color | PPA level: 0.0883 -> 0.0003 | Hurt |
| increase beta from 0 to 1 (ablation) | F^beta · Alpaca-GPT4 (expertise) | win rate: 0.7613 -> 0.7636 | No clear effect |
| increase beta from 0 to 1 (ablation) | F^beta · Alpaca-GPT4 (style) | PPA level: 0.5012 -> 0.3635 | Hurt |
Closest papers by meaning
- No Preference Left Behind: Group Distributional Preference Optimization (ICLR 2025 · similarity 0.89)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.88)
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.88)
- Pairwise Calibrated Rewards for Pluralistic Alignment (NeurIPS 2025 · similarity 0.87)
- Selective Preference Aggregation (ICML 2025 · similarity 0.87)
- Preference Learning Algorithms Do Not Learn Preference Rankings (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
