Research map / Large language models / Alignment and preferences
Causally Robust Preference Learning with Reasons
Minjune Hwang, Yigit Korkmaz, Daniel Seita, Erdem Biyik
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reason-orthogonal consistency loss (ablation) | ReCouPLe · ManiSkill 2-task OOD | mean OOD reward accuracy: 0.726 -> 0.872 | Improved |
| add reason-orthogonal consistency loss (ablation) | ReCouPLe · ManiSkill 4-task OOD | mean OOD reward accuracy: 0.745 -> 0.878 | Improved |
| add reward-ratio regularizer on top of consistency removal (ablation) | ReCouPLe · ManiSkill OOD | reward accuracy: 0.727 -> 0.726 | No clear effect |
| add rationale projection with equality constraint (ReCouPLe-EC) | BT-Multi · ManiSkill Pick-Larger 2-task OOD | reward accuracy: 0.600 -> 0.820 | Improved |
| add naive reason auxiliary loss (RFP) | BT-Multi · ManiSkill Pick-Larger 2-task OOD | reward accuracy: 0.600 -> 0.620 | No clear effect |
| use inequality constraint (IC) instead of equality (EC) | ReCouPLe · ManiSkill Pick-Larger 2-task OOD | reward accuracy: 0.820 -> 0.633 | Hurt |
| use inequality constraint (IC) instead of equality (EC) | ReCouPLe · Meta-World Push-Wall visual OOD | reward accuracy: 0.760 -> 0.900 | Improved |
| add ReCouPLe reasons for transfer to novel task | BT-Multi · Meta-World Pick-Place novel | reward accuracy: 0.547 -> 0.663 | Improved |
| add ReCouPLe-EC to visual confounder | BT · Meta-World Push visual OOD | reward accuracy: 0.233 -> 0.713 | Improved |
| use diversified paraphrased rationales (16 per task) (ablation) | ReCouPLe-EC · ManiSkill Pick OOD | reward accuracy: 0.820 -> 0.867 | Improved |
| provide rationales for only 25% of pairs (ablation) | ReCouPLe-EC · ManiSkill Pick OOD | reward accuracy: 0.540 -> 0.783 | Improved |
| increase preference queries from 1000 to 2000 per task (ablation) | ReCouPLe-EC · ManiSkill Pick OOD | reward accuracy: 0.820 -> 0.913 | Improved |
Closest papers by meaning
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.92)
- CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries (ICML 2025 · similarity 0.88)
- Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input (ICML 2024 · similarity 0.88)
- Bayesian Preference Learning for Test-Time Steerable Reward Models (NeurIPS 2026 · similarity 0.88)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.88)
- ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
