Research map / Large language models / Reasoning and chain of thought
Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Chain-of-Preference Optimization (CPO), which builds per-step preference pairs from ToT search trees and fine-tunes CoT reasoning with DPO, matching ToT accuracy at CoT latency.
Problem
Chain-of-thought reasoning follows a single path and can miss better reasoning paths that tree-of-thought finds, but ToT's search makes inference far too slow.
Approach
Use the intermediate thoughts generated and pruned during ToT search as per-step preference pairs (selected vs. sibling thoughts), and train the LLM with a step-wise DPO objective so plain CoT decoding reproduces ToT-preferred reasoning without search at inference.
Key results
- CPO improves CoT accuracy by an average of 4.3% across models and tasks, with a maximum gain of 9.7%.
- CPO averages 40.9% on LLaMA2-7B versus 36.0% for CoT and 39.1% for ToT, while running about 57.5x faster than ToT on average.
- FPO (full-path DPO) yields a 4.6% relative drop versus CPO and performs worse than SFT, which the authors attribute to LCP gradient cancellation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CPO rather than CoT | LLaMA2-7B · Average Performance | accuracy: 36.0 -> 40.9 | Helped |
| Use CPO rather than TS-SFT | LLaMA2-13B · Average Performance | accuracy: 43.1 -> 45.2 | Helped |
| Use CPO rather than CoT | Mistral-7B · Average Performance | accuracy: 45.0 -> 49.1 | Helped |
| Use CPO instead of ToT decoding | LLaMA2-7B · Average Performance | latency: 1749.1 -> 37.9 | Helped |
| Use CPO instead of TS-SFT | LLaMA2-7B · Average Performance | accuracy: 38.3 -> 40.9 | Helped |
| Train CPO on uniform QA data rather than single-task data (ablation) | LLaMA2-7B CPO · Bamboogle | accuracy: 32.0 -> 35.2 | Helped |
| Train CPO on mixed-type data rather than single-task data (ablation) | LLaMA2-7B CPO · Bamboogle | accuracy: 32.0 -> 35.2 | Helped |
| Increase CPO-only training from one to two iterations (ablation) | LLaMA2-7B · Bamboogle with CoT inference | accuracy: 32.0 -> 33.6 | Helped |
| Increase CPO-only training from two to three iterations (ablation) | LLaMA2-7B · Bamboogle with CoT inference | accuracy: 33.6 -> 31.2 | Hurt |
| Use ToT inference after initial SFT rather than base-model ToT inference (ablation) | LLaMA2-7B SFT+CPO · Bamboogle | accuracy: 33.6 -> 30.4 | Hurt |
| Use FPO at full-path level instead of per-step CPO (ablation) | LLaMA2-7B · Bamboogle | accuracy: -4.6% relative performance decrease | Hurt |
| Increase the proportion of dispreferred thoughts used in optimization (ablation) | CPO · Bamboogle | accuracy | Helped |
Surprising findings
- Full-path preference optimization (FPO) underperforms plain SFT, attributed to longest-common-prefix gradient cancellation.
- After fine-tuning, ToT performance on the tuned model can decline, possibly because output diversity drops and shrinks the search space.
Limitations
- Preference data generation requires running ToT, which is time-consuming.
- Evaluated only on text LLMs and a narrow set of tasks (QA, fact verification, arithmetic), not vision-language models.
- Test sets were capped at 300 samples per dataset due to ToT's computational cost.
Experimental setup
Datasets: Bamboogle WikiMultiHopQA HotpotQA FEVER FEVEROUS VitaminC SVAMP
Benchmarks: Bamboogle WikiMultiHopQA HotpotQA FEVER FEVEROUS VitaminC SVAMP
Metrics: accuracy latency (s/instance) F1
Baselines: CoT ToT TS-SFT ReST self-rewarding FPO
Builds on: chain-of-thought prompting tree-of-thought direct preference optimization TS-SFT ReST self-rewarding
Closest papers by meaning
- C3oT: Generating Shorter Chain-of-Thought Without Compromising Effectiveness (AAAI 2025 · similarity 0.91)
- Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization (AAAI 2026 · similarity 0.91)
- Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning (ICML 2025 · similarity 0.91)
- Chain-of-Thought Reasoning Without Prompting (NeurIPS 2024 · similarity 0.90)
- Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization (ICLR 2026 · similarity 0.90)
- Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.