Research map / Large language models / Alignment and preferences
Optimal Design for Human Preference Elicitation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes the Kiefer-Wolfowitz optimal design to lists of K items to compute a fixed sampling policy over lists for eliciting absolute or ranking human feedback (Dope), with prediction-error and ranking-loss bounds and experiments on synthetic and Nectar/Anthropic QA data.
Problem
Efficiently eliciting human preference feedback under a query budget so that a preference model can correctly rank K answers to each of L questions, with absolute or ranking feedback.
Approach
A matrix generalization of the Kiefer-Wolfowitz theorem lets the G-optimal design be computed over lists (matrices of item feature vectors) rather than individual vectors; the policy samples lists proportionally to the design, and the estimator is OLS (absolute) or the Zhu et al. PL MLE (ranking).
Key results
- On the Nectar subset, Dope reaches the best baseline's lowest ranking loss (500 interactions) with 150 interactions, more than a three-fold reduction in sample size.
- On the Anthropic subset, Dope matches the best baseline's ranking loss at n=1000 with n=300 interactions.
- Theory: maximum prediction error is O~(d^2/n) for absolute feedback and O(K^6 d^2/n) for ranking feedback; ranking loss decays exponentially in n and squared gaps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Dope's design with a plug-in design (ablation) | Dope · Synthetic ranking feedback (K=2, n=500) | maximum prediction error: 15.79 ± 1.08 -> 19.75 ± 1.48 (Plug-in (400)); 30.52 ± 3.00 (Plug-in (300)); 65.75 ± 13.71 (Plug-in (200)); 100.39 ± 10.7 | Hurt |
| Replace Dope's design with a plug-in design (ablation) | Dope · Synthetic ranking feedback (K=2, n=500) | ranking loss: 0.107 ± 0.002 -> 0.104 ± 0.003 (Plug-in (400)); 0.103 ± 0.002 (Plug-in (300)); 0.114 ± 0.003 (Plug-in (200)); 0.142 ± 0. | Mixed |
| Use the D-optimal design computed with the unknown θ* instead of Dope's implementable design (ablation) | Dope · Synthetic ranking feedback (K=2, n=500) | maximum prediction error: 15.79 ± 1.08 -> 9.22 ± 0.82 | Helped |
| Use the D-optimal design computed with the unknown θ* instead of Dope's implementable design (ablation) | Dope · Synthetic ranking feedback (K=2, n=500) | ranking loss: 0.107 ± 0.002 -> 0.092 ± 0.002 | Helped |
| Increase the number of lists L (ablation) | Dope · Synthetic ranking feedback; K=2, 3, 4, and 5 | ranking loss | Hurt |
| Increase list length K (ablation) | Dope · Synthetic ranking feedback; L=50, 100, 200, and 500 | ranking loss | Helped |
| Use Dope's optimal list design rather than the listed baselines | Preference elicitation · Synthetic absolute feedback | ranking loss | Helped |
| Use Dope's optimal list design rather than the listed baselines | Preference elicitation · Synthetic ranking feedback | ranking loss: n = 100 -> n = 60 | Helped |
| Use Dope's optimal list design rather than the listed baselines | Preference elicitation · Nectar dataset | ranking loss: n = 500 -> n = 150 | Helped |
| Use Dope's optimal list design rather than the listed baselines | Preference elicitation · Anthropic dataset | ranking loss: n = 1 000 -> n = 300 | Helped |
Surprising findings
- Plug-in designs with an estimated theta did not show clear benefits over worst-case designs in the logistic experiment, so the authors did not adopt them.
Limitations
- Designs are non-adaptive and may not be optimal for ranking loss.
- No matching lower bounds are derived.
- Experiments use simulated feedback and small question sets, not real LLM reward model training.
Experimental setup
Datasets: Nectar Anthropic HH dataset Yahoo! Front Page Today Module (feature preprocessing reference)
Benchmarks: Synthetic absolute feedback Synthetic ranking feedback Nectar subset Anthropic subset
Metrics: ranking loss (Kendall tau pair count divided by L) maximum prediction error
Baselines: Unif Avg-Design Clustered-Design APO Plug-in design
Builds on: Kiefer-Wolfowitz theorem Zhu et al. (2023) offline RLHF ranking estimator APO (Das et al. 2024) Plackett-Luce model
Closest papers by meaning
- Enhancing Preference-based Linear Bandits via Human Response Time (NeurIPS 2024 · similarity 0.87)
- Eliciting Human Preferences with Language Models (ICLR 2025 · similarity 0.87)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.86)
- Comparing Few to Rank Many: Active Human Preference Learning Using Randomized Frank-Wolfe Method (ICML 2025 · similarity 0.86)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.86)
- Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.