Research map / Large language models / Alignment and preferences
Token-Importance Guided Direct Preference Optimization
Ning Yang, Hai Lin, Yibo Liu, Baoliang Tian, Guoqing Liu, Haijun Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DPO with TI-DPO (token weights + triplet loss) | DPO · six-benchmark average | avg score: 57.7 -> 62.3 | Improved |
| TI-DPO vs GRPO on average | TI-DPO · six-benchmark average | avg score: 62.3 vs 62.1 | No clear effect |
| Add triplet loss (ablation) | TI-DPO · Llama-3.2-3B-Instruct ablation | Math: 79.0 -> 80.7 | Improved |
| Add triplet loss (ablation) | TI-DPO · Llama-3.2-3B-Instruct ablation | Reliability: 83.0 -> 86.8 | Improved |
| Use gradient-attribution importance weights instead of uniform weights (ablation) | TI-DPO · Llama-3.2-3B-Instruct ablation | Reasoning: 30.5 -> 34.6 | Improved |
| Use gradient-attribution weights instead of random weights (ablation) | TI-DPO · Llama-3.2-3B-Instruct ablation | Instr-Follow: 55.0 -> 63.5 | Improved |
| Add Gaussian prior to weights (ablation) | TI-DPO · Llama-3.2-3B-Instruct ablation | Reliability: 82.5 -> 86.8 | Improved |
| Use Gaussian prior instead of softmax prior (ablation) | TI-DPO · Llama-3.2-3B-Instruct ablation | Reliability: 81.0 -> 86.8 | Improved |
| Use TI-DPO instead of DPO under 40% label noise | DPO · label-noise robustness | accuracy: 60.1 -> 68.3 | Improved |
| Use TI-DPO instead of DPO | DPO · generation diversity | S-BLEU: 34.2% -> 30.1% | Improved |
| Use TI-DPO instead of GRPO on reasoning-heavy tasks | GRPO · GSM8K | accuracy: 75.7 -> 73.0 | Hurt |
Closest papers by meaning
- TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights (ICLR 2025 · similarity 0.93)
- Token-level Direct Preference Optimization (ICML 2024 · similarity 0.91)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization (ICML 2025 · similarity 0.90)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.90)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
