Research map / Large language models / Alignment and preferences
Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pragmatic feature preference queries that augment example-level pairwise comparisons with feature-level preferences and language-derived irrelevant-feature data augmentation to learn linear reward models more efficiently.
Problem
Reward models learned from example-level pairwise preferences (RLHF) lose fine-grained information about which features matter and why, making reward learning inefficient with limited comparisons.
Approach
Enrich binary preference queries with feature-level preferences and language descriptions of reward-relevant features; infer that unmentioned features are irrelevant and synthesize new preference pairs by swapping their values, with a joint loss combining example-level and feature-level Bradley-Terry terms.
Key results
- On simulated mushroom foraging, the pragmatic feature preference method (FPPrag) converges to more accurate rewards with fewer examples than RLHF, with the gap largest for sparse reward functions.
- On flight booking with real human language descriptions parsed by GPT-4, the method outperforms traditional RLHF with fewer examples across 20 reward functions.
- In a 36-participant user study, FPPrag queries did not significantly increase perceived effort, frustration, or time versus RLHF queries (e.g., frustration t(11)=-0.87, p=0.19).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine feature-level and example-level comparisons with pragmatic augmentation | Reward learning · Mushroom foraging | probability assigned to ground-truth best examples | Helped |
| Combine feature-level and example-level comparisons with pragmatic augmentation | Reward learning · Flight booking | probability assigned to ground-truth best examples | Helped |
| Combine feature-level and example-level comparisons with pragmatic augmentation | Reward learning · Mushroom foraging, real user responses | probability assigned to ground-truth best examples | Helped |
| Add feature-level comparisons to example-level comparisons (ablation) | Pragmatic feature preference learning · Mushroom foraging | probability assigned to ground-truth best examples | Helped |
| Add pragmatic augmentation of comparisons using linguistic descriptions (ablation) | Pragmatic feature preference learning · Mushroom foraging | probability assigned to ground-truth best examples | Helped |
| Use example-level comparisons only, without feature-level comparisons or pragmatic augmentation (ablation) | Reward learning · Mushroom foraging | probability assigned to ground-truth best examples | Hurt |
| Add feature-level comparisons to example-level comparisons (ablation) | Pragmatic feature preference learning · Mushroom foraging, sparse reward features | probability assigned to ground-truth best examples | Helped |
| Use pragmatic feature preference queries rather than example-only queries | Reward learning with real users · Mushroom foraging user study | perceived challenge describing preferences: t(11)=0.16, p=0.43 | No clear effect |
| Use pragmatic feature preference queries rather than example-only queries | Reward learning with real users · Mushroom foraging user study | frustration with providing labels: t(11)=-0.87, p=0.19 | No clear effect |
| Use pragmatic feature preference queries rather than feature-only queries | Reward learning with real users · Mushroom foraging user study | time spent on task: t(11)= -0.24, p=0.41 | No clear effect |
| Use feature-only queries rather than example-only queries (ablation) | Reward learning with real users · Mushroom foraging user study | perceived challenge describing preferences: t(11)=2.31, p=0.02 | Hurt |
| Use feature-only queries rather than example-only queries (ablation) | Reward learning with real users · Mushroom foraging user study | time spent on task: t(11)=-1.86, p=0.03 | Hurt |
Surprising findings
- Feature-level preference queries (FP) without pragmatic augmentation significantly increased perceived difficulty of describing preferences and time on task compared to RLHF queries.
Limitations
- Assumes irrelevant features can be swapped freely, which is hard for text-based domains.
- Assumes language descriptions can be grounded to the correct state features; GPT-4 parsing of feature maps was not validated for robustness.
- Experiments are on linear reward models in small discrete bandit settings with 20 flight reward functions and 5 seeds.
Experimental setup
Datasets: Mushroom foraging simulated task (own) Flight booking human descriptions (Lin et al. rewards-from-language dataset) Mushroom foraging user study responses (Prolific, 36 participants)
Benchmarks: mushroom foraging (simulated) flight booking (human language) mushroom foraging user study
Metrics: probability assigned to ground-truth best example Likert survey ratings (effort, frustration, communication) time on task
Baselines: RLHF (example-level comparisons) feature preference (FP) ablation pragmatic comparison (RLHF-prag) ablation
Builds on: RLHF / Bradley-Terry pairwise preference loss linear contextual bandits rewards-from-language (Lin et al.)
Closest papers by meaning
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.90)
- Rectifying Shortcut Behaviors in Preference-based Reward Learning (NeurIPS 2025 · similarity 0.90)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.89)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.89)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.88)
- Preference Learning with Response Time: Robust Losses and Guarantees (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.