Research map / Large language models / Reasoning and chain of thought

DEPO: Dual-Efficiency Preference Optimization for LLM Agents

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines dual-efficiency (tokens per step and steps per trajectory) for LLM agents and proposes DEPO, an efficiency-bonus extension of KTO that rewards succinct, short successful trajectories using offline labels.

Problem

LLM agents produce long chains of thought and take many interaction steps, hurting efficiency in real-world use, and there is no systematic definition of agent efficiency or training method targeting it.

Approach

Defines dual-efficiency (step-level tokens and trajectory-level steps) and adds a parameter-independent efficiency bonus to the desirable log-ratio in KTO-style preference optimization, applied only to successful trajectories, so efficient successful trajectories get a larger preference margin without paired data or on-policy sampling.

Key results

• Llama3.1-8B-BC+DEPO reduces T@All on WebShop by 24.6% vs BC, and on BabyAI T@All falls 60.9% vs BC (as reported for the combined comparisons).
• Qwen2.5-7B-BC+DEPO boosts BabyAI success and reward by 29.3% and 21.1% over vanilla KTO.
• DEPO retains efficiency gains when trained on only 25% of the data, with T@All improving more than 10% over BC.

Surprising findings

• Applying an equally strong efficiency penalty to undesirable trajectories does not improve results and generally degrades performance and efficiency.
• Several large instruct models (e.g., Qwen2.5-72B-Instruct, DeepSeek-V3) achieve very low success rates on WebShop and BabyAI under the 0-shot ReAct prompt.

Limitations

• Bonus applied only to successful trajectories; efficiency gains rely on hand-set alpha hyperparameters tuned per model.
• Evaluated on only two agent environments plus math benchmarks; data generated with a single strong teacher (DeepSeek-V3) via MCTS.

Experimental setup

Datasets: WebShop BabyAI GSM8K MATH SimulEq

Benchmarks: WebShop BabyAI GSM8K MATH SimulEq

Metrics: success rate mean reward tokens per trajectory steps per trajectory accuracy

Baselines: DeepSeek-V3 DeepSeek-R1-Distill-Qwen-7B DeepSeek-R1-Distill-Llama-8B Llama-3.1-8B-Instruct Llama-3.2-3B-Instruct Qwen2.5-3B-Instruct Qwen2.5-7B-Instruct Qwen2.5-14B-Instruct Qwen2.5-72B-Instruct Behavioral cloning KTO Token Budget

Builds on: KTO DPO ReAct Monte Carlo Tree Search LoRA

Closest papers by meaning

• Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.90)
• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
• MallowsPO: Fine-Tune Your LLM with Preference Dispersions (ICLR 2025 · similarity 0.89)
• DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization (NeurIPS 2025 · similarity 0.89)
• Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents (ICLR 2026 · similarity 0.89)
• MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems (ICML 2026 · similarity 0.89)