AI Paper Atlas

Research map / Large language models / Alignment and preferences

When Labelers Stay Silent: The Power of Ties in Cost-Effective Preference Learning

Jiaqi Lv, Zihan Zhang, Chengjing Yang, Shiyu Xia, Ning Xu, Xin Geng

ICML 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
filter out tie pairs from GPT-4 golden labels (ablation)DPO · UltraFeedback, Llama-3-8BGold Reward: 6.48 -> 6.78Improved
filter out tie pairs from GPT-4 golden labels (ablation)DPO · UltraFeedback, Mistral-7BGold Reward: 4.40 -> 5.13Improved
filter out ties from Llama-3B relabeled data (ablation)DPO · UltraFeedback, Llama-3-8BGold Reward: 7.94 -> 8.06Improved
model ties with symmetric tie loss instead of forced choice (ablation)DPO with Llama-3B labeler · UltraFeedback, Llama-3-8BGold Reward: 7.94 -> 8.20Improved
use Llama-3B silent-aware labels instead of GPT-4 forced labelsDPO · UltraFeedback, Llama-3-8BGold Reward: 6.48 -> 8.20Improved
add quality tags (Both-Good/Both-Bad) with directional tie loss and corrected risk (ablation)SymTie · UltraFeedback, Llama-3B labeler delta_M=1, Llama-3-8BGold Reward: 8.20 -> 9.24Improved
add deconstruct-then-KTO on quality-tagged ties (ablation)SymTie · UltraFeedback, Llama-3B labeler delta_M=1, Llama-3-8BGold Reward: 8.20 -> 8.81Improved
use SymTie instead of TODO tie-aware baselinetie handling · UltraFeedback, Llama-3B labeler delta_M=1, Llama-3-8BGold Reward: 8.28 -> 8.20No clear effect
increase resolution threshold delta_M (more ties) (ablation)silent-aware methods · UltraFeedback, Llama-3B/8B labelersGold Reward: e.g. Llama-8B SymTie 8.23 (delta_M=1) -> 10.28 (delta_M=5)Improved
increase tie ratio to 80% percentile threshold (ablation)SymTie and C-DirTie with Qwen labelers · Llama-3-8B policyGold ScoreHurt
apply tie modeling to human-labeled data (ChatArena golden)Forced DPO · ChatArena, Llama-3-8BGold Reward: 6.73 -> 9.99Improved
silent-aware objectives on Mistral-7B with ChatArena (ablation)Filtered baseline · ChatArena, Mistral-7B, Llama-3B delta_M=5Gold Reward: Forced 7.15, Filtered 5.06, SymTie 7.29, C-DirTie 6.52Mixed

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy