Research map / Large language models / Alignment and preferences

TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees

ICLR 2025 · Large language models › Alignment and preferences · approach: DPO design choices · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TPO, a preference-list ranking objective with lambda weights and semantic-similarity-based adaptive step rewards that trains LLMs directly on entire ToT preference trees for math reasoning, outperforming DPO across five open models.

Problem

DPO only accepts binary preference pairs, so sampling pairs from ToT preference trees wastes dispreferred responses, ignores graded rewards among failures, and yields small reward margins when trajectories share sub-trajectories.

Approach

Treat alignment as Preference List Ranking (LambdaLoss-style, with rewards from ChatGPT scores) over all tree branches, and reweight per-step implicit reward margins by cosine similarity of step embeddings (Adaptive Step Reward).

Key results

• Qwen2-7B-Instruct+TPO averages 54.43 on math vs 50.89 for DPO and 45.23 for the base model.
• Qwen2-1.5B-Instruct+TPO (32.22 avg) surpasses Qwen2-7B-Instruct (45.23 avg) on math tasks in the table, and DeepSeekMath-7B-Instruct+TPO reaches 73.36 avg vs 73.10 for DeepSeekMath-7B-RL.
• On Llama-3-8B-Instruct, TPO averages 60.36 on math vs 56.40 for DPO.

Measured effects reported in the paper

Surprising findings

• On HumanEval, TPO and DPO decreased performance for Qwen2 models, which the authors attribute to catastrophic forgetting.
• On MMLU, all alignment methods perform similarly, with TPO giving no gain.

Limitations

• TPO may cause stronger catastrophic forgetting, degrading out-of-distribution performance such as HumanEval.
• Imbalanced reward distribution in the preference tree due to ChatGPT scoring tendencies and the data generation pipeline.
• Reliance on ChatGPT scoring with a reference trajectory as the reward signal.

Experimental setup

Datasets: MetaMath MMIQC AQuA MATH SVAMP ASDiv GSM-Plus HumanEval MBPP BBH MMLU

Benchmarks: MATH SVAMP ASDiv GSM-Plus MBPP HumanEval BBH MMLU

Metrics: pass@1 accuracy

Baselines: SFT DPO DPO* DPO+ DeepSeekMath-7B-RL GPT-3.5 Turbo GPT-4 S-DPO Rose-DPO ODPO

Builds on: Direct Preference Optimization Tree-of-Thoughts LambdaLoss Self-Consistency Chain-of-Thought

Closest papers by meaning

• Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.91)
• Language Models as Implicit Tree Search (ICML 2025 · similarity 0.91)
• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.91)
• Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.91)
• Advancing LLM Reasoning Generalists with Preference Trees (ICLR 2025 · similarity 0.90)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization (ICLR 2025 · similarity 0.90)