AI Paper Atlas

Research map / Large language models / Alignment and preferences

PORT: Preference Optimization via Robust Token-Level Reweighting

Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu, Xueru Zhang, Mohammad Mahdi Khalili

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PORT, a robust preference optimization method that reweights preference learning at the token level to handle noisy annotations instead of treating whole responses as uniformly correct or incorrect.

Problem

Preference optimization for LLM alignment degrades when preference annotations are noisy from annotation errors and ambiguity; existing robust methods operate at the sequence level, treating entire responses as uniformly correct or incorrect.

Approach

Moves robust preference optimization from the sequence level to the token level by reweighting individual tokens' contributions so that noisy portions of preferred and dispreferred responses are down-weighted.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use PORT token-level CDF reweighting instead of the strongest sequence-level baseline (beta-DPO)DPO · AlpacaFarm LLaMA2-7BGold RM score: 3.42 -> 3.82Improved
use PORT instead of DPODPO · AlpacaFarm LLaMA2-7BGPT win rate (%): 56.8 -> 67.6Improved
use PORT instead of SamSSamS · AlpacaFarm Qwen3-4BGPT win rate (%): 53.8 -> 53.6No clear effect
use PORT instead of SamSSamS · AlpacaFarm Qwen3-4BGold RM score: 2.84 -> 2.98Improved
use PORT instead of SamSSamS · Anthropic HH harmlessPKU-Beaver safety rate: 0.53 -> 0.55Improved
increase label-flip noise from p=0.2 to p=0.4DPO · Anthropic HH harmlessPKU-Beaver safety rate: 0.45 -> 0.38Hurt
increase label-flip noise from p=0.2 to p=0.4PORT · Anthropic HH harmlessPKU-Beaver safety rate: 0.56 -> 0.53Hurt
use PORT instead of strongest baseline as noise growsbeta-DPO · Anthropic HH harmless, p=0.2 vs 0.4SR gap: 0.02 -> 0.06 (table); text says 0.03 -> 0.06Improved
use PORT instead of Dr.DPO under epsilon-mislabeled noiseDr.DPO · Anthropic HH, p=0.4PKU-Beaver safety rate: 0.48 -> 0.53Improved
use PORT instead of Dr.DPO under alpha-uncertain noiseDr.DPO · Anthropic HH, p=0.4PKU-Beaver safety rate: 0.44 -> 0.49Improved
compute CDF weights from a single preference pair (B=1) instead of B=16 (ablation)PORT · Qwen2.5-0.5B Anthropic HHGPT win rate (%): 44.3 -> 39.1Hurt
increase batch size from 4 to 16 (ablation)PORT · Qwen2.5-0.5B Anthropic HHGPT win rate (%): 42.8 -> 44.3No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy