Research map / Large language models / Alignment and preferences
TODO: Enhancing LLM Alignment with Ternary Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TOBT, a tie-aware extension of the Bradley-Terry model, and TODO, a DPO-style objective that learns from ternary (prefer/tie/disprefer) preference labels, improving preference modeling and benchmark alignment for Mistral-7B and Llama 3-8B.
Problem
Binary Bradley-Terry based preference optimization such as DPO cannot handle tied preference pairs, which are common in preference datasets (e.g., 12-29% tie rates) and often lead to biased updates from noisy or uninformative pairs.
Approach
Introduce a margin alpha into the BT model's integration so that a tie is the event where the strength difference falls within a band; this yields a ternary TOBT model and a DPO-style loss that uses a prefer-loss for non-tied pairs and a tie-loss (with zero update at zero reward margin) for tied pairs.
Key results
- On Mistral-7B, TODO (tie ratio 0.2) reaches 71.17 average over six OpenCompass benchmarks versus 70.78 for DPO at the same tie ratio, and 71.02 vs 66.24 for binary DPO vs TODO at tie ratio 0.
- On Reward Bench, Mistral+TODO averages 73.64 (tie ratio 0) vs 71.03 for DPO; Llama 3 TODO reaches 76.81 at tie ratio 0.1 vs 74.45 for DPO.
- On Ultrafeedback with tie ratio 0.2, Mistral-7B TODO test-set accuracy is 76.40 vs 75.00 for DPO; on Chatarena with tie ratio 0.17, TODO achieves 78.47 vs 76.33 for DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TODO instead of DPO with tie ratio 0.0 | Mistral-7B · Downstream benchmarks | Average: 66.24 -> 71.02 | Helped |
| Use TODO instead of DPO with tie ratio 0.1 | Mistral-7B · Downstream benchmarks | Average: 70.58 -> 71.06 | Helped |
| Use TODO instead of DPO with tie ratio 0.2 | Mistral-7B · Downstream benchmarks | Average: 70.78 -> 71.17 | Helped |
| Use TODO instead of DPO with tie ratio 0.3 | Mistral-7B · Downstream benchmarks | Average: 65.57 -> 67.37 | Helped |
| Use TODO instead of DPO with tie ratio 0.0 | Llama 3-8B · Downstream benchmarks | Average: 70.75 -> 71.03 | Helped |
| Use TODO instead of DPO with tie ratio 0.1 | Llama 3-8B · Downstream benchmarks | Average: 70.68 -> 70.86 | Helped |
| Use TODO instead of DPO with tie ratio 0.2 | Llama 3-8B · Downstream benchmarks | Average: 70.42 -> 70.98 | Helped |
| Use TODO instead of DPO with tie ratio 0.3 | Llama 3-8B · Downstream benchmarks | Average: 70.52 -> 71.13 | Helped |
| Use TODO instead of DPO at tie ratio 0.0 | Mistral-7B · Reward Bench | Average accuracy: 71.03 -> 73.64 | Helped |
| Use TODO instead of DPO at tie ratio 0.1 | Llama 3-8B · Reward Bench | Average accuracy: 74.45 -> 76.81 | Helped |
| Use TODO instead of DPO with tie ratio 0.2 | Mistral-7B · Ultrafeedback preference test | Accuracy: 75.00 -> 76.40 | Mixed |
| Use TODO instead of DPO with tie ratio 0.17 | Mistral-7B · Chatarena preference test | Accuracy: 76.33 -> 78.47 | Helped |
Surprising findings
- DPO's test accuracy and MT Bench sometimes drop when tie data is added, while TODO with binary-only data also outperforms DPO, suggesting gains are not solely from tie data.
- Large margin values (alpha above about 1) cause divergence or degraded performance, so alpha must be kept small.
Limitations
- Tie labels in the main experiments are derived from equal Ultrafeedback quality scores, a heuristic that may not reflect true human ties.
- Hyperparameter alpha selection relies on heuristic loss thresholds and empirical sweeps, with divergence for large alpha.
- Extensions to RLHF reward models and online policies are proposed but not evaluated.
Experimental setup
Datasets: Ultrafeedback Ultrafeedback binarized Ultrachat-200k Chatarena Reward Bench
Benchmarks: Reward Bench MT Bench Piqa ARC-c ARC-e MMLU Hellaswag Winogrande
Metrics: preference prediction accuracy MT Bench score accuracy
Baselines: DPO KTO SimPO ODPO SFT
Builds on: Direct Preference Optimization Bradley-Terry model ODPO Ultrafeedback
Closest papers by meaning
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.94)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.93)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.93)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.