Research map / Reinforcement learning / Visual and representation RL
Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLIDER, a parameter-efficient two-level LLM agent (shared LoRA backbone with high- and low-level prompts) trained via SFT, then offline hierarchical actor-critic RL, and adapted online by freezing low-level skills.
Problem
LLM agents struggle with long-horizon decision-making under sparse rewards due to poor exploration and long-term credit assignment; RL-based fine-tuning is sample-inefficient.
Approach
Shares one frozen LLM backbone (LoRA actor, MLP critic) between a high-level planner that emits subtask goals every c steps and a low-level controller that executes them with intrinsic completion rewards, trained with offline sentence-level actor-critic (IQL-style expectile value, AWAC-style weighted MLE actor) and then adapted online by fine-tuning only the high-level policy.
Key results
- On ScienceWorld unseen tasks with Llama-3-8B, GLIDER scores 68.34 versus 52.33 for the best baseline ETO (reported +30.59%).
- On ALFWorld unseen with Mistral-7B, GLIDER reaches 74.83 versus 71.43 for ETO.
- Ablation on Llama-3B: hierarchical SFT+ORL reaches 61.29 on ScienceWorld unseen, above non-hierarchical Mistral-7B at 58.50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GLIDER instead of the strongest reported baseline, ETO | Mistral-7B · ScienceWorld Seen | score: 58.17 -> 67.31 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Mistral-7B · ScienceWorld Unseen | score: 51.85 -> 65.14 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Mistral-7B · ALFWorld Seen | score: 66.84 -> 70.02 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Mistral-7B · ALFWorld Unseen | score: 71.43 -> 74.83 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Gemma-7B · ScienceWorld Seen | score: 50.44 -> 63.67 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Gemma-7B · ScienceWorld Unseen | score: 47.84 -> 58.50 | Helped |
| use GLIDER instead of NAT | Gemma-7B · ALFWorld Seen | score: 67.86 -> 72.12 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Gemma-7B · ALFWorld Unseen | score: 68.66 -> 70.88 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Llama-3-8B · ScienceWorld Seen | score: 57.90 -> 77.43 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Llama-3-8B · ScienceWorld Unseen | score: 52.33 -> 68.34 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Llama-3-8B · ALFWorld Seen | score: 64.29 -> 71.56 | Helped |
| use GLIDER instead of the strongest reported baseline, ETO | Llama-3-8B · ALFWorld Unseen | score: 64.18 -> 75.38 | Helped |
Surprising findings
- Offline RL trained from scratch outperforms SFT-only agents in the ablations, which the authors link to DeepSeek-R1's observations on reinforcement fine-tuning.
- Medium-quality data alone (score 36.0) outperforms expert-only data (29.7) on unseen ScienceWorld tasks.
Limitations
- Multi-stage training pipeline (SFT, offline RL, online adaptation) is complex.
- Low-level policy parameters cannot be strictly frozen due to parameter sharing, so they are maintained by continued offline training.
- Evaluated only on two text-based interactive benchmarks; not tested on math or code.
Experimental setup
Datasets: ScienceWorld ALFWorld
Benchmarks: ScienceWorld ALFWorld
Metrics: score (seen) score (unseen)
Baselines: ReAct Reflexion SwiftSage NAT ETO AC AWAC
Builds on: implicit Q-learning AWAC ETO ReAct LoRA
Closest papers by meaning
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.90)
- Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning (ICLR 2026 · similarity 0.90)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.89)
- $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.89)
- The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data (ICML 2025 · similarity 0.89)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.