Research map / Multi-agent systems and games / LLM agents

Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Hierarchical Preference Learning (HPL), combining trajectory-, step-, and action-group-level DPO with a dual-layer (group length × reward-gap) curriculum for offline preference optimization of LLM agents.

Problem

Preference-based DPO for long-horizon LLM agents suffers a granularity mismatch: trajectory-level DPO gives coarse credit assignment, while step-level DPO is noisy and data-inefficient under limited Monte Carlo rollouts.

Approach

Decomposes expert trajectories into action groups (sub-tasks via fixed, entropy-based, or LLM-semantic segmentation), builds group-level preference pairs with MC-estimated rewards, and trains with a summed trajectory + step + group DPO loss under a 3-phase curriculum over group length and reward gap.

Key results

• HPL (Semantic) on Qwen2.5-7B-Instruct reaches average 67.28 vs. 63.82 for IPR and 63.47 for ETO across ALFWorld, WebShop, and InterCode-SQL.
• On ALFWorld unseen with Qwen2.5-7B, HPL (Semantic) achieves 84.08% success vs. 78.11% for IPR.
• Removing the curriculum (HPL Static) lowers the 7B average from 67.81 to 65.30 in the ablation table.

Surprising findings

• Simple fixed-length segmentation (Fixed-N, Fixed-K) already outperforms ETO and IPR, suggesting intermediate granularity helps even without semantic boundaries.

Limitations

• Performance depends on segmentation quality; the semantic variant needs a powerful external LLM (GPT-4o).
• The curriculum adds hyperparameters (length and reward-gap thresholds) that may be domain-specific.
• Reliance on a teacher model (GPT-4o) means the policy may inherit teacher biases.

Experimental setup

Datasets: ALFWorld WebShop InterCode-SQL Spider

Benchmarks: ALFWorld WebShop InterCode-SQL

Metrics: success rate average reward

Baselines: SFT RFT ETO IPR GPT-4o Gemini-2.5-Pro

Builds on: DPO ETO IPR RFT

Closest papers by meaning

• Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.90)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.90)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.90)
• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
• AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.89)
• DEPO: Dual-Efficiency Preference Optimization for LLM Agents (AAAI 2026 · similarity 0.89)