Research map / Probabilistic and scientific modeling / Causal discovery
Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training
Christian Moya, Alex Semendinger, Guang Lin, Elliott Thornley
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add informative tie training (equal-utility pairs with differing spurious features) to DPO | DPO on synthetic hotel preferences · Hotel preferences, adversarial shift (Qadv) | pairwise accuracy (%): 64.20% -> 86.70% | Improved |
| add informative tie training to DPO | DPO on synthetic hotel preferences · Hotel preferences, suppressed correlation (Qsup) | pairwise accuracy (%): 74.00% -> 82.85% | Improved |
| add informative tie training to DPO | DPO on synthetic hotel preferences · Hotel preferences, in-distribution (P) | pairwise accuracy (%): 92.25% -> 92.40% | No clear effect |
| add tie training (alpha = 0.75) to DPO on an MLP scorer | DPO with MLP scorer on synthetic features · Neural network adversarial accuracy | adversarial accuracy: ≈0.25 -> ≈0.7 | Improved |
| add tie training (alpha = 0.75) to DPO on an MLP scorer | DPO with MLP scorer on synthetic features · Neural network spurious gap | spurious gap (aligned minus misaligned accuracy): ≈0.7 -> ≈0.45 | Improved |
| increase fraction of strict preferences (reduce tie mixing) (ablation) | tie training MLP scorer · Neural network spurious gap | spurious gap | Hurt |
| increase training sample size n under strict DPO (ablation) | strict DPO on log-linear model · Linear deployment suboptimality | deployment suboptimality | No clear effect |
| increase training samples under strict DPO (ablation) | strict DPO on MLP scorer · Neural network adversarial accuracy | adversarial accuracy | No clear effect |
| use ties at small sample size (ablation) | tie training MLP scorer · Neural network small n | adversarial accuracy | No clear effect |
Closest papers by meaning
- Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.88)
- On Extending Direct Preference Optimization to Accommodate Ties (NeurIPS 2025 · similarity 0.87)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.86)
- KL Penalty Control via Perturbation for Direct Preference Optimization (NeurIPS 2025 · similarity 0.86)
- Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspective (ICML 2025 · similarity 0.86)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
