Research map / Large language models / Alignment and preferences
DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning
Yifan Wang, Bolian Li, Junlin Wu, Zhaoxuan Tan, Zheli Liu, Ruqi Zhang, Ananth Grama, Qingkai Zeng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DRIFT instead of IterDPO | Qwen2.5-7B-Instruct · WildFeedback controlled iter1 WildBench | Task Score: 47.07 -> 51.06 | Improved |
| use DRIFT instead of SPIN | Qwen2.5-7B-Instruct · WildFeedback controlled iter1 WildBench | Task Score: 42.86 -> 51.06 | Improved |
| run a second SPIN iteration | SPIN · WildFeedback controlled 7B WildBench | Task Score: 42.86 -> 37.86 | Hurt |
| train DRIFT on DSAT negatives with fresh policy positives vs base | Qwen2.5-14B-Instruct · UltraFeedback WildBench | Task Score: 55.08 -> 59.27 | Improved |
| use DRIFT instead of base | Qwen2.5-7B-Instruct · AlpacaEval2 WildFeedback controlled | win rate: 37.69 -> 43.79 | Improved |
| use full 11k DSAT data instead of controlled 4k for IterDPO | IterDPO iter2 · WildFeedback 7B WildBench | Task Score: 48.94 -> 46.17 | Hurt |
| use full 11k DSAT data instead of controlled 4k for DRIFT | DRIFT iter2 · WildFeedback 7B WildBench | Task Score: 51.06 -> 51.69 | Improved |
| remove guidance prompt (DSAT reference and improvement instruction) from DRIFT positive generation (ablation) | DRIFT iter1 · WildFeedback full 7B WildBench | Task Score: 50.61 -> 50.77 | No clear effect |
| add guidance prompt to IterDPO (ablation) | IterDPO iter1 · WildFeedback full 7B WildBench | Task Score: 49.44 -> 46.31 | Hurt |
| extend DRIFT from iter4 to iter5 | DRIFT · WildFeedback 7B five iterations | WildBench Task Score: 52.47 -> 51.22 | No clear effect |
| use DRIFT instead of IterDPO on Gemma-3-12B-it | Gemma-3-12B-it · WildFeedback WildBench | Task Score: 48.98 -> 60.88 | Improved |
| train with DRIFT instead of base | Qwen2.5-7B-Instruct · ToxiGen | toxic score: 1.56 -> 1.58 | No clear effect |
Closest papers by meaning
- Aligning Language Models Using Follow-up Likelihood as Reward Signal (AAAI 2025 · similarity 0.88)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.88)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.88)
- Preference Learning with Response Time: Robust Losses and Guarantees (NeurIPS 2025 · similarity 0.87)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.87)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
