Research map / Reinforcement learning / RL from feedback
CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLARIFY, an offline PbRL method that learns a preference-aware trajectory embedding via contrastive losses and uses rejection sampling over embedding distances to select less ambiguous queries for non-ideal (skip-based) teachers.
Problem
Human preference labels between very similar trajectory segments are ambiguous, which lowers label efficiency and limits real-world use of offline preference-based RL.
Approach
Learn a trajectory embedding with an ambiguity loss (push apart clearly distinguished pairs, pull together no_cop pairs) and a quadrilateral loss (preference-consistent distance ordering), then select queries by rejection sampling on embedding distances so that clearly distinguished pairs are favored.
Key results
- Under skip rate 0.5, CLARIFY achieves the best success rate on most Metaworld tasks, e.g., dial-turn 77.50 vs 57.33 for OPRL and drawer-open 83.50 vs 72.67 for OPRL.
- Clearly-distinguished query ratio on dial-turn rises to 76.33% for CLARIFY versus 46.95% for MR and 31.67% for OPRL.
- With real human labelers on walker-walk, CLARIFY reaches 420.75 episodic return vs 265.91 for OPRL, with label accuracy 87.08% vs 66.67%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLARIFY instead of OPRL with real human feedback | Offline PbRL on walker-walk · walker-walk, real human labels | episodic return: 265.91 ± 33.57 -> 420.75 ± 52.02 | Helped |
| Use CLARIFY instead of MR | Offline PbRL · dial-turn, skip rate 0.5; 1000 queries | success rate: 57.33 ± 5.02 -> 77.50 ± 3.01 | Helped |
| Add the ambiguity loss to the model without either contrastive loss (ablation) | CLARIFY · dial-turn, skip rate 0.5 | success rate: 63.20 ± 4.79 -> 69.00 ± 11.20 | Helped |
| Add the ambiguity loss to the model without either contrastive loss (ablation) | CLARIFY · sweep-into, skip rate 0.5 | success rate: 40.00 ± 11.29 -> 52.80 ± 17.01 | Helped |
| Add the quadrilateral loss to the model without either contrastive loss (ablation) | CLARIFY · dial-turn, skip rate 0.5 | success rate: 63.20 ± 4.79 -> 71.25 ± 8.81 | Helped |
| Add the quadrilateral loss to the model without either contrastive loss (ablation) | CLARIFY · sweep-into, skip rate 0.5 | success rate: 40.00 ± 11.29 -> 62.20 ± 4.92 | Helped |
| Add the quadrilateral loss when using the ambiguity loss (ablation) | CLARIFY · dial-turn, skip rate 0.5 | success rate: 69.00 ± 11.20 -> 77.50 ± 3.01 | Helped |
| Add the quadrilateral loss when using the ambiguity loss (ablation) | CLARIFY · sweep-into, skip rate 0.5 | success rate: 52.80 ± 17.01 -> 68.00 ± 3.19 | Helped |
| Replace rejection sampling with highest-density rho_clr query selection (ablation) | CLARIFY · dial-turn, skip rate 0.5 | success rate: 77.50 ± 3.01 -> 61.33 ± 5.31 | Hurt |
| Replace rejection sampling with highest-density rho_clr query selection (ablation) | CLARIFY · sweep-into, skip rate 0.5 | success rate: 68.00 ± 3.19 -> 48.60 ± 10.74 | Hurt |
| Use CLARIFY query selection instead of the listed baselines | Offline PbRL with non-ideal teacher · clearly distinguished query ratio across dial-turn, hammer, sweep-into, and walker-walk; skip rate 0.5 | query clarity ratio | Helped |
| Use CLARIFY-selected queries instead of OPRL-selected queries | Human preference labeling on walker-walk · walker-walk, real human labels | query clarity ratio: 53.33 ± 6.24 -> 63.33 ± 8.50 | Helped |
Surprising findings
- Density-based selection of the most clearly distinguishable queries performed worse than rejection sampling, attributed to reduced query diversity.
- Using the ambiguity loss alone gave unstable results due to early overfitting.
Limitations
- Evaluated only on a small set of Metaworld and DMControl tasks with a scripted skip-based teacher whose design is the authors' own.
- Human experiments involve only the authors as labelers and a small number of queries (20-100 per run).
Experimental setup
Datasets: LiRE offline dataset (Metaworld, DMControl) scripted teacher preference labels human preference labels from authors
Benchmarks: Metaworld (7 tasks) DMControl (cheetah-run, walker-walk)
Metrics: success rate episodic return query clarity ratio label accuracy
Baselines: Markovian Reward (MR) Preference Transformer (PT) OPRL OPPO LiRE IQL with ground-truth reward
Builds on: Preference Transformer OPPO LiRE Bi-directional Decision Transformer IQL
Closest papers by meaning
- Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries (ICML 2025 · similarity 0.92)
- Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.89)
- Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options (NeurIPS 2025 · similarity 0.89)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.