Research map / Large language models / Alignment and preferences
PORT: Preference Optimization via Robust Token-Level Reweighting
Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu, Xueru Zhang, Mohammad Mahdi Khalili
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PORT, a robust preference optimization method that reweights preference learning at the token level to handle noisy annotations instead of treating whole responses as uniformly correct or incorrect.
Problem
Preference optimization for LLM alignment degrades when preference annotations are noisy from annotation errors and ambiguity; existing robust methods operate at the sequence level, treating entire responses as uniformly correct or incorrect.
Approach
Moves robust preference optimization from the sequence level to the token level by reweighting individual tokens' contributions so that noisy portions of preferred and dispreferred responses are down-weighted.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PORT token-level CDF reweighting instead of the strongest sequence-level baseline (beta-DPO) | DPO · AlpacaFarm LLaMA2-7B | Gold RM score: 3.42 -> 3.82 | Improved |
| use PORT instead of DPO | DPO · AlpacaFarm LLaMA2-7B | GPT win rate (%): 56.8 -> 67.6 | Improved |
| use PORT instead of SamS | SamS · AlpacaFarm Qwen3-4B | GPT win rate (%): 53.8 -> 53.6 | No clear effect |
| use PORT instead of SamS | SamS · AlpacaFarm Qwen3-4B | Gold RM score: 2.84 -> 2.98 | Improved |
| use PORT instead of SamS | SamS · Anthropic HH harmless | PKU-Beaver safety rate: 0.53 -> 0.55 | Improved |
| increase label-flip noise from p=0.2 to p=0.4 | DPO · Anthropic HH harmless | PKU-Beaver safety rate: 0.45 -> 0.38 | Hurt |
| increase label-flip noise from p=0.2 to p=0.4 | PORT · Anthropic HH harmless | PKU-Beaver safety rate: 0.56 -> 0.53 | Hurt |
| use PORT instead of strongest baseline as noise grows | beta-DPO · Anthropic HH harmless, p=0.2 vs 0.4 | SR gap: 0.02 -> 0.06 (table); text says 0.03 -> 0.06 | Improved |
| use PORT instead of Dr.DPO under epsilon-mislabeled noise | Dr.DPO · Anthropic HH, p=0.4 | PKU-Beaver safety rate: 0.48 -> 0.53 | Improved |
| use PORT instead of Dr.DPO under alpha-uncertain noise | Dr.DPO · Anthropic HH, p=0.4 | PKU-Beaver safety rate: 0.44 -> 0.49 | Improved |
| compute CDF weights from a single preference pair (B=1) instead of B=16 (ablation) | PORT · Qwen2.5-0.5B Anthropic HH | GPT win rate (%): 44.3 -> 39.1 | Hurt |
| increase batch size from 4 to 16 (ablation) | PORT · Qwen2.5-0.5B Anthropic HH | GPT win rate (%): 42.8 -> 44.3 | No clear effect |
Closest papers by meaning
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.91)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.91)
- RePO: Understanding Preference Learning Through ReLU-Based Optimization (NeurIPS 2025 · similarity 0.90)
- Distributionally Robust Listwise Preference Optimization (NeurIPS 2026 · similarity 0.90)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.90)
- ROPO: Robust Preference Optimization for Large Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
