Research map / Large language models / Alignment and preferences
Language Models as Implicit Tree Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IT-PO, an RL-free preference optimization method in which a second LM learns an implicit tree search policy asymptotically equivalent to AlphaZero-like MCTS, improving human preference alignment and LLM reasoning on GSM8K, Game24, and MATH.
Problem
DPO lacks the search capability of MCTS-based reasoning, while standard MCTS for LMs relies on RL-learned value functions that conflict with DPO's RL-free nature.
Approach
Treats AlphaZero-like MCTS as a regularized (reverse-KL) stochastic policy, and trains a separate LM policy πφ to approximate this implicit tree search policy via a DPO-style preference loss, with no explicit value or reward model; decoding and training are self-improved through preference policy augmentation.
Key results
- On Anthropic HH, IT-PO φ reaches 69.12% accuracy versus 59.43% for DPO and 67.33% for TDPO2.
- On Game24 with Path@1, ITS-Rollout reaches 73.2% versus 71.3% for MCTS-Rollout at comparable token budgets.
- On MATH with Qwen1.5-32B, ITS-rollout reaches 40.2 versus 36.7 for MCTS-rollout and 36.1 for greedy CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IT-PO φ instead of TDPO2 | Pythia 2.8 preference alignment · Anthropic HH | accuracy (%): 67.33 -> 69.12 | Helped |
| Use IT-PO φ instead of TDPO2 | Pythia 2.8 preference alignment · Anthropic HH | entropy: 4.915 -> 5.315 | Helped |
| Use ITS-α instead of MCTS-α | Llama-2-7B reasoning · GSM8K Path@1 | accuracy (%): 51.9 -> 53.2 | Helped |
| Use ITS-Rollout instead of MCTS-Rollout | Llama-2-7B reasoning · Game24 Path@1 | accuracy (%): 71.3 -> 73.2 | Helped |
| Use ITS-Rollout instead of MCTS-Rollout | Llama-2-7B reasoning · GSM8K Path@1 | accuracy (%): 47.8 -> 51.6 | Helped |
| Use ITS-Rollout instead of MCTS-Rollout | Qwen1.5-32B mathematical reasoning · MATH | accuracy (%): 36.7 -> 40.2 | Helped |
| Increase tree width from 6 to 50 (ablation) | MCTS-α · Game24 Path@1 | accuracy (%): 41.6 -> 74.5 | Helped |
| Increase tree width from 6 to 50 (ablation) | MCTS-Rollout · Game24 Path@1 | accuracy (%): 43.8 -> 80.7 | Helped |
| Increase tree width from 6 to 50 (ablation) | BFS-V · Game24 Path@1 | accuracy (%): 43.2 -> 74.6 | Helped |
| Increase tree width from 6 to 50 (ablation) | ITS-α · Game24 Path@1 | accuracy (%): 46.1 -> 75.0 | Helped |
| Increase tree width from 6 to 50 (ablation) | ITS-Rollout · Game24 Path@1 | accuracy (%): 48.9 -> 81.8 | Helped |
| Use LLAMA3-8B rather than Qwen1.5-32B for φ (ablation) | IT-PO mathematical reasoning · MATH | accuracy (%): 39.8 -> 37.9 | Hurt |
Surprising findings
- On GSM8K, tree-search baselines (excluding the proposed method) do not clearly outperform standard CoT, and MCTS variants underperform BFS-V despite higher compute.
Limitations
- Requires a second LM φ, increasing computational and memory cost over single-LM DPO variants.
- Gains on GSM8K are limited; the MCTS-derived strategy is unreliable on MATH according to the authors' analysis.
- Theoretical equivalence to AlphaZero-like search holds only asymptotically with many simulations.
Experimental setup
Datasets: Anthropic HH-RLHF GSM8K MATH Game24
Benchmarks: Anthropic HH GSM8K Game24 MATH
Metrics: accuracy entropy (diversity) win rate Path@1 token count
Baselines: DPO RTO TDPO1 TDPO2 CoT-greedy BFS-V MCTS-alpha MCTS-Rollout
Builds on: DPO AlphaZero Implicit tree search (Grill et al., 2020) cDPO MCTS-alpha
Closest papers by meaning
- TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.90)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.90)
- Bootstrapping Language Models with DPO Implicit Rewards (ICLR 2025 · similarity 0.90)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.90)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.