Research map / Learning theory and algorithms / Prediction and decision losses
TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization
Chengtao Jian, Kai Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TPO, a tri-level distributionally robust learning framework for direct preference optimization that targets out-of-distribution preference shift across group, prompt, and conditional-preference levels.
Problem
DPO minimizes empirical risk over observed preference comparisons, assuming the training preference distribution matches the deployment distribution; this i.i.d. assumption often fails under hierarchical preference distribution shift (group-level, prompt-level, and conditional preference shift).
Approach
Models preference distribution shift as a natural three-level hierarchy and applies tri-level distributionally robust optimization to DPO so that the learned policy is robust to shifts at each level rather than fitting the empirical preference distribution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine group, prompt and preference robustness (full TPO) instead of any single level (ablation) | TPO · Emotion OOD combined shift, Qwen2.5-0.5B | average preference accuracy: 84.37 -> 85.25 (best single level, Prompt, vs full) | Improved |
| add prompt-level adversarial embedding perturbation only (ablation) | DPO · Emotion OOD combined shift, Qwen2.5-0.5B | average preference accuracy: 82.60 -> 84.37 | Improved |
| add group-level reweighting only (ablation) | DPO · Emotion OOD combined shift, Qwen2.5-0.5B | average preference accuracy: 82.60 -> 82.06 | No clear effect |
| add preference-level margin penalty only (ablation) | DPO · Emotion OOD combined shift, Qwen2.5-0.5B | average preference accuracy: 82.60 -> 83.24 | Improved |
| use full TPO instead of DPO | LLaMA-3.2-1B · Emotion OOD combined shift | average preference accuracy: 87.44 -> 88.31 | Improved |
| use full TPO instead of DPO on preference-shift worst-group accuracy | LLaMA-3.2-1B · Emotion OOD preference shift | worst-group preference accuracy: 81.42 -> 81.31 | No clear effect |
| add TPO robustness to DPO | LLaMA-3.1-8B · Open LLM Leaderboard v2 IFEval | score: 75.4 -> 79.6 | Improved |
| add TPO robustness to DPO | LLaMA-3.2-3B · Open LLM Leaderboard v2 IFEval | score: 61.5 -> 65.6 | Improved |
| add TPO robustness to DPO | LLaMA-3.2-1B/3B/8B · Open LLM Leaderboard v2 MMLU-Pro and GPQA | score: MMLU-Pro 1B 20.0 -> 20.5 | No clear effect |
| add TPO robustness to DPO | LLaMA-3.2-3B · Open LLM Leaderboard v2 MATH-Hard | score: 21.4 -> 21.1 (rho 0) | Hurt |
| vary preference-level radius (ablation) | TPO · Open LLM Leaderboard v2 | score: 8B IFEval 77.6 / 79.6 / 77.5 | Mixed |
| add TPO penalty on large margins | DPO · Preference poisoning, margin-based flip, high poisoning rate | preference accuracy | Improved |
Experimental setup
Builds on: DPO
Closest papers by meaning
- Self-Improving Robust Preference Optimization (ICLR 2025 · similarity 0.86)
- MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge (NeurIPS 2026 · similarity 0.86)
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.86)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.86)
- Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.86)
- Provably Robust DPO: Aligning Language Models with Noisy Feedback (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
