Research map / Large language models / Reasoning and chain of thought
Advancing LLM Reasoning Generalists with Preference Trees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UltraInteract, a tree-structured multi-turn reasoning alignment dataset with paired correct/incorrect actions, and the Eurus LLM suite (7B and 70B) trained with SFT and preference learning (DPO/KTO/NCA), plus Eurus-RM-7B with a new reward-modeling objective.
Problem
Open-source LLMs lag proprietary models on complex reasoning, partly due to scarce high-quality alignment data and underexplored preference learning for reasoning tasks.
Approach
Collect per-instruction preference trees of multi-turn, tool-integrated, critique-refined reasoning actions with ground-truth-verified correct/incorrect pairs; find DPO hurts reasoning while KTO/NCA help (chosen-reward trend matters), and add an absolute-reward term to the Bradley-Terry reward loss.
Key results
- Eurus-70B achieves 33.3% pass@1 on LeetCode and 32.6% on TheoremQA, beating GPT-3.5 Turbo on reasoning overall (avg 59.0 for Eurus-70B+NCA vs 57.0 for GPT-3.5 Turbo).
- Eurus-RM-7B achieves better correlation with human annotators than all compared models on AutoJ and MT-Bench (MT-Bench 72.9 vs GPT-4 63.9), and 82.4 RewardBench avg.
- DPO hurts performance on most benchmarks and training of the 70B DPO model fails as rewards go to negative infinity; KTO and NCA improve results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add KTO preference learning to Eurus-7B-SFT | Eurus-7B-SFT · Math benchmarks and multi-turn evaluation | performance | Helped |
| Add NCA preference learning to Eurus-70B-SFT | Eurus-70B-SFT · TheoremQA | pass@1 accuracy: 28.0 -> 32.6 | Helped |
| Add DPO preference learning to Eurus-7B-SFT | Eurus-7B-SFT · Most reasoning benchmarks | performance: 20.0 -> 8.3 (LeetCode) | Hurt |
| Remove the direct-reward objective L_DR (ablation) | Eurus-RM-7B · RewardBench Reasoning | score: 87.0 -> 77.5 | Hurt |
| Remove the Bradley-Terry objective L_BT (ablation) | Eurus-RM-7B · RewardBench Chat-Hard | score: 65.3 -> 58.5 | Hurt |
| Replace generated actions with ground-truth rationales and answers (ablation) | Eurus-7B-SFT · Coding average | score: 44.9 -> 33.9 | Hurt |
| Train using open-source alignment data only, without UltraInteract (ablation) | Eurus-7B-SFT · Math average | score: 58.5 -> 33.5 | Hurt |
| Train using UltraInteract only (ablation) | Eurus-7B-SFT · IFEval | loose score: 44.0 -> 17.4 | Hurt |
| Train using UltraInteract only (ablation) | Eurus-7B-SFT · BBH | pass@1 accuracy: 64.6 -> 67.0 | Helped |
| Remove UltraSafety from reward-model training data (ablation) | Eurus-RM-7B · Reasoning reranking | pass@1 accuracy | Helped |
| Remove UltraFeedback and UltraSafety from reward-model training data (ablation) | Eurus-RM-7B · AutoJ Overall | correlation: 64.5 -> 58.7 | Hurt |
| Remove either L_BT or L_DR (ablation) | Eurus-RM-7B · HumanEval reranking as response count increases from 8 to 16 | pass@1 accuracy | Mixed |
Surprising findings
- DPO degrades reasoning performance on most benchmarks despite working for general chat, while KTO and NCA help.
- Training only on UltraInteract data hurts instruction following substantially (IFEval 17.4 vs 44.0).
Limitations
- Ground-truth-based correctness limits preference pairs to tasks with reference solutions.
- Lower instruction-following and general knowledge (MMLU) than general-purpose models; 70B variant trails on knowledge tasks due to CodeLLaMA base.
- Some benchmarks (MBPP, MATH, TheoremQA) have contamination issues in baselines that were struck through.
Experimental setup
Datasets: UltraInteract UltraChat ShareGPT OpenOrca UltraFeedback UltraSafety GSM8K MATH MathQA NumGLUE TabMWP CodeContest
Benchmarks: HumanEval MBPP LeetCode GSM-Plus MATH TheoremQA SVAMP ASDiv BBH-Hard IFEval MINT MMLU
Metrics: pass@1 prompt-level loose score success rate at turn 5 accuracy correlation with human annotators
Baselines: Mistral-7B-Instruct-v0.2 Zephyr-7B-beta OpenChat-3.5-1210 Starling-LM-7B-alpha DeepSeek-Coder-33B-Instruct DeepSeek-LM-67B-Chat QWen1.5-72B-Chat CodeLLaMA-70B-Instruct OpenCI-CL-70B OpenMath-CL-70B GPT-3.5 Turbo GPT-4
Builds on: DPO KTO NCA Bradley-Terry reward modeling UltraFeedback OpenCodeInterpreter
Closest papers by meaning
- TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.90)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
- Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees (NeurIPS 2024 · similarity 0.89)
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.89)
- Language Models as Implicit Tree Search (ICML 2025 · similarity 0.89)
- Building Math Agents with Multi-Turn Iterative Preference Learning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.