Research map / Large language models / Alignment and preferences
Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
Anirudh Bharadwaj, Chaitanya Malaviya, Nitish Joshi, Mark Yatskar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune reward models with counterfactual data augmentation (CDA) | reward models · All five biases (avg) | miscalibration rate (%): 39.4 -> 32.5 | Improved |
| fine-tune reward models with CDA | reward models · All five biases (avg) | absolute skew difference (%): 20.5 -> 10.0 | Improved |
| fine-tune with CDA for vagueness counterfactuals | reward models · Vagueness | miscalibration: -22.8 | Improved |
| fine-tune with CDA for jargon counterfactuals | reward models · Jargon | miscalibration: -17.1 | Improved |
| fine-tune with CDA for length counterfactuals | reward models · Length | miscalibration: -3.4 | No clear effect |
| fine-tune with CDA for structure counterfactuals | reward models · Structure | miscalibration: 12.6 -> 17.3 | Hurt |
| fine-tune with CDA for sycophancy counterfactuals | reward models · Sycophancy | miscalibration: 40.6 -> 44.4 | Hurt |
| add pooled length, jargon and vagueness counterfactuals in one fine-tuning run (ablation) | reward models · Length skew | skew rate (%): 60.1 -> 45.2 | Improved |
| add pooled multi-bias counterfactuals (ablation) | reward models · Vagueness skew | skew rate (%): 61 -> 38.5 | Improved |
| fine-tune with CDA | reward models · RewardBench | average score | No clear effect |
Closest papers by meaning
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.91)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.91)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.90)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.90)
- Diverging Preferences: When do Annotators Disagree and do Models Know? (ICML 2025 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
