Research map / Reinforcement learning / RL from feedback
Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes distinguishability queries (DQ/DistQ) in which labelers pick the easier of two pairwise comparisons and then answer only that one, combined with variance/entropy-based query selection and a joint loss for RLHF reward learning on simulated control tasks.
Problem
Pairwise trajectory comparisons in RLHF are hard for humans when trajectories differ subtly and convey only ordinal information, whereas RL needs cardinal utilities (preference strength).
Approach
A distinguishability query presents two candidate pairwise comparisons; the labeler chooses which is easier to distinguish and answers only that one, yielding both a strength signal and an ordinal label. Selection filters queries by ensemble variance (informativeness) then by prediction entropy (easiness), and a joint loss couples distinguishability and pairwise feedback.
Key results
- On Walker walk with a 200-query budget, DQ (half budget, 100 distinguishability queries) and DQ (full budget) outperform the PEBBLE, SURF, RUNE, MRN and QPA baselines in episode reward, with QPA slightly ahead of DQ on Humanoid stand.
- In a single-participant user study on a Quadruped stand-and-wave task, the agent trained with DistQ succeeded in 10/10 evaluation rounds versus 0/10 for PEBBLE.
- DistQ makes the fewest incorrectly predicted pairwise feedback among compared methods on Walker walk and Quadruped walk, indicating easier queries for the labeler.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use distinguishability queries instead of standard pairwise comparison queries | DistQ · DMControl locomotion and Meta-World manipulation tasks | episode reward or success rate | Helped |
| Use half the query budget with distinguishability queries | DistQ · DMControl locomotion and Meta-World manipulation tasks | episode reward or success rate | Mixed |
| Use distinguishability queries rather than standard pairwise comparisons | DistQ · Quadruped stand-and-wave user study | success rate: 0% -> 100% | Helped |
| Select informative queries first, then favor easier queries (ablation) | DistQ · Walker walk and Quadruped walk | episode reward | Helped |
| Prioritize easiness before informativeness (ablation) | DistQ · Walker walk and Quadruped walk | episode reward | Hurt |
| Select queries based only on informativeness (ablation) | DistQ · Walker walk and Quadruped walk | episode reward | Helped |
| Select queries based only on easiness (ablation) | DistQ · Walker walk and Quadruped walk | episode reward | Hurt |
| Pair PCQs by matching easier queries with less easy queries rather than pairing randomly (ablation) | DistQ · Quadruped walk and Sweep into | episode reward or success rate | Helped |
| Remove distinguishability preference loss and train with pairwise loss only (ablation) | DistQ · Walker walk and Quadruped walk | episode reward | Hurt |
| Use DistQ-selected queries with pairwise loss only instead of PEBBLE's queries (ablation) | DistQ · Walker walk and Quadruped walk | episode reward | Helped |
Surprising findings
- DistQ trained only on the pairwise loss still outperforms PEBBLE, suggesting its selected queries are more effective even without the distinguishability loss.
- Prioritizing easiness before informativeness (EI variant) substantially degrades reward performance.
Limitations
- Evaluation limited to simulated control tasks with a scripted labeler; only one real human in the user study.
- Informativeness and easiness measures are heuristic and could be refined.
- Not yet applied to language modeling or real-world applications.
Experimental setup
Datasets: DMControl Meta-World
Benchmarks: DMControl Walker walk DMControl Quadruped walk DMControl Humanoid stand Meta-World Window open Meta-World Door unlock Meta-World Door open Meta-World Sweep into Meta-World Disassemble
Metrics: episode reward success rate number of wrongly predicted pairwise feedback user study success rate
Baselines: PEBBLE SURF RUNE MRN QPA SAC
Builds on: PEBBLE SAC Bradley-Terry model
Closest papers by meaning
- CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries (ICML 2025 · similarity 0.92)
- ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning (NeurIPS 2025 · similarity 0.91)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.90)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.90)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.