Research map / Large language models / Alignment and preferences
DuPO: Enabling Reliable Self-Verification via Dual Preference Optimization
Shuaijie She, Yu Bao, Yu Lu, Lu Xu, Tao Li, Wenhao Zhu, Jianbing Zhang, Shujian Huang, Shanbo Cheng, Lu Lu, Yuxuan Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply DuPO RL training | Seed-X-7B-Instruct · Flores 756 directions | COMET: 86.96 -> 89.09 | Improved |
| apply DuPO RL training | Qwen3-4B · AMC23/AIME24/AIME25 average | accuracy: 77.2 -> 83.6 | Improved |
| apply DuPO RL training | DeepSeek-R1-Distill-Qwen-1.5B · AMC23/AIME24/AIME25 average | accuracy: 35.8 -> 39.7 | Improved |
| apply DuPO RL training | OpenReasoning-Nemotron-7B · AIME24 | accuracy: 83.3 -> 83.3 | No clear effect |
| apply DuPO RL training | OpenReasoning-Nemotron-7B · AIME25 | accuracy: 73.3 -> 90.0 | Improved |
| apply DuPO instead of SimpleRL-Zoo (oracle labels) | LlaMA-3.1-8B · AMC23/MATH500 average | accuracy: 19.0 -> 32.1 | Improved |
| apply DuPO RL training | OctoThinker-8B-Hybrid-Base · AMC23/MATH500 average | accuracy: 23.8 -> 62.5 | Improved |
| apply DuPO directly to base model without SFT | Qwen3-4B-Base · AMC23 | accuracy: 20% -> 70% | Improved |
| use DuPO backward accuracy for inference-time reranking | Qwen3-4B · AIME24/AIME25 average | accuracy: 68.4 -> 77.7 | Improved |
| use DuPO backward accuracy for inference-time reranking | DeepSeek-R1-Distill-Qwen-1.5B · AIME24/AIME25 average | accuracy: 20.0 -> 38.7 | Improved |
| add unknown component selection / dual question filtering (ablation) | DuPO on DeepSeek-R1-Distill-Qwen-1.5B · math benchmarks | accuracy: 3.6 point drop without filtering | Improved |
| add unknown component selection / dual question filtering (ablation) | DuPO on Qwen3-4B · math benchmarks | accuracy: +5.4 points on average over poorer duality variant | Improved |
Closest papers by meaning
- DEPO: Dual-Efficiency Preference Optimization for LLM Agents (AAAI 2026 · similarity 0.88)
- One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.87)
- Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards (ICML 2026 · similarity 0.87)
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.87)
- $f$-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment (NeurIPS 2026 · similarity 0.87)
- DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
