AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization

Chengtao Jian, Kai Yang

NeurIPS 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Distributionally robust optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TPO, a tri-level distributionally robust learning framework for direct preference optimization that targets out-of-distribution preference shift across group, prompt, and conditional-preference levels.

Problem

DPO minimizes empirical risk over observed preference comparisons, assuming the training preference distribution matches the deployment distribution; this i.i.d. assumption often fails under hierarchical preference distribution shift (group-level, prompt-level, and conditional preference shift).

Approach

Models preference distribution shift as a natural three-level hierarchy and applies tri-level distributionally robust optimization to DPO so that the learned policy is robust to shifts at each level rather than fitting the empirical preference distribution.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
combine group, prompt and preference robustness (full TPO) instead of any single level (ablation)TPO · Emotion OOD combined shift, Qwen2.5-0.5Baverage preference accuracy: 84.37 -> 85.25 (best single level, Prompt, vs full)Improved
add prompt-level adversarial embedding perturbation only (ablation)DPO · Emotion OOD combined shift, Qwen2.5-0.5Baverage preference accuracy: 82.60 -> 84.37Improved
add group-level reweighting only (ablation)DPO · Emotion OOD combined shift, Qwen2.5-0.5Baverage preference accuracy: 82.60 -> 82.06No clear effect
add preference-level margin penalty only (ablation)DPO · Emotion OOD combined shift, Qwen2.5-0.5Baverage preference accuracy: 82.60 -> 83.24Improved
use full TPO instead of DPOLLaMA-3.2-1B · Emotion OOD combined shiftaverage preference accuracy: 87.44 -> 88.31Improved
use full TPO instead of DPO on preference-shift worst-group accuracyLLaMA-3.2-1B · Emotion OOD preference shiftworst-group preference accuracy: 81.42 -> 81.31No clear effect
add TPO robustness to DPOLLaMA-3.1-8B · Open LLM Leaderboard v2 IFEvalscore: 75.4 -> 79.6Improved
add TPO robustness to DPOLLaMA-3.2-3B · Open LLM Leaderboard v2 IFEvalscore: 61.5 -> 65.6Improved
add TPO robustness to DPOLLaMA-3.2-1B/3B/8B · Open LLM Leaderboard v2 MMLU-Pro and GPQAscore: MMLU-Pro 1B 20.0 -> 20.5No clear effect
add TPO robustness to DPOLLaMA-3.2-3B · Open LLM Leaderboard v2 MATH-Hardscore: 21.4 -> 21.1 (rho 0)Hurt
vary preference-level radius (ablation)TPO · Open LLM Leaderboard v2score: 8B IFEval 77.6 / 79.6 / 77.5Mixed
add TPO penalty on large marginsDPO · Preference poisoning, margin-based flip, high poisoning ratepreference accuracyImproved

Experimental setup

Builds on: DPO

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy