Research map / Large language models / Alignment and preferences
Robust Preference Alignment via Directional Neighborhood Consensus
Ruochen Mao, Yuling Shi, Xiaodong Gu, Jiaheng Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sample candidates from neighboring preference directions instead of the target direction | DPA-v1-Mistral-7B · UltraFeedback, 45 deg | RPS win rate (%): 53.4 (20 deg) -> 69.1 (45 deg) | Improved |
| neighborhood sampling (RPS) on SFT model | Mistral-7B-Instruct-v0.2 · HelpSteer2, 45 deg | RPS win rate (%): 52.1 (10 deg) -> 94.3 (45 deg) | Improved |
| neighborhood sampling (RPS) on DPO model | Zephyr-7B-Beta · UltraFeedback (mean) | RPS win rate (%): 52.1 | Improved |
| increase neighborhood size k from 3 to 5 (ablation) | RPS on DPA · HelpSteer2 (v3-v10) | RPS win rate (0-1): 0.543 -> 0.608 | Improved |
| increase neighborhood size k from 3 to 5 (ablation) | RPS on SFT · HelpSteer2 (v3-v10) | RPS win rate (0-1): 0.564 -> 0.673 | Improved |
| increase neighborhood size k from 3 to 5 (ablation) | RPS on DPO · HelpSteer2 (v3-v10) | RPS win rate (0-1): 0.494 -> 0.535 | Improved |
| use angular radius 30 deg instead of 20 deg (ablation) | RPS on SFT · HelpSteer2 (v3-v10) | RPS win rate (0-1): 0.557 -> 0.673 | Improved |
| use angular radius 30 deg instead of 40 deg (ablation) | RPS on SFT · HelpSteer2 (v3-v10) | RPS win rate (0-1): 0.559 -> 0.673 | Improved |
| use angular radius 30 deg instead of 40 deg (ablation) | RPS on DPA · HelpSteer2 (v3-v10) | RPS win rate (0-1): 0.579 -> 0.608 | Improved |
| use angular radius 20 deg instead of 30 deg (ablation) | RPS on DPO · HelpSteer2 (v3-v10) | RPS win rate (0-1): 0.535 -> 0.485 | Hurt |
Closest papers by meaning
- HPS: Hard Preference Sampling for Human Preference Alignment (ICML 2025 · similarity 0.93)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.93)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.92)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.92)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.92)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
