Research map / Large language models / Agents, code and math
Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IALP, which augments user instructions into PDDL problems using promptable predicates (via GPT-4o) and grounding-mechanism predicates (voxel map, LangSAM, GraspNet, reachability map) to drive closed-loop LLM planning for long-horizon mobile manipulation on a real Tiago++ robot.
Problem
Long-horizon mobile manipulation planning for humanoid/mobile robots in real-world unstructured environments requires grounded knowledge (e.g., object positions, grasp feasibility), which LLM planners relying on textual descriptions or prompt engineering lack.
Approach
Augment user instructions into PDDL problem states using both LLM-prompted (promptable) predicates and grounding-mechanism predicates computed from perception (navigation path feasibility, detection, grasp and placement feasibility, reachability), then let the LLM choose among K candidate actions by token probability in a closed loop with replanning at each step.
Key results
- IALP achieves over 80% success rate on all five long-horizon real-world tasks under the ablation setup with GPT-4o as planner.
- Removing feasibility feedback substantially lowers success rate because the planner cannot generate feasible actions without find/reachable predicates.
- Across 100 trials, 15 predicate-checking errors were recorded, categorized as planning, promptable, or grounding failures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add grounding-based feasibility feedback to the PDDL description (ablation) | IALP with GPT-4o · Five long-horizon tasks | executable-action success rate | Helped |
| Remove optimal action selection (ablation) | IALP with GPT-4o · Five long-horizon tasks | executable-action success rate | No clear effect |
Surprising findings
- Removing optimal selection (token-probability choice among candidates) still keeps a relatively high success rate, since feasibility checks already filter actions.
Limitations
- Depends on a pre-defined primitive action library, restricting range of instructions and generalization.
- Real-world hardware noise and physical limitations cause predicate and execution failures.
- Promptable predicates rely on VLM reasoning which is limited for precise spatial information.
Experimental setup
Benchmarks: real-world Tiago++ long-horizon tasks (Place box, Take jacket, Take pillbox, Insert book, Lift bucket)
Metrics: success rate token probability of action candidates navigation/manipulation/planning time predicate failure counts
Baselines: IALP w/o Feasibility Feedback IALP w/o Optimal Selection
Builds on: PDDL OK-Robot LangSAM GraspNet VoxPoser
Closest papers by meaning
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.90)
- R2C: Mapping Room to Chessboard to Unlock LLM As Low-Level Action Planner (CVPR 2025 · similarity 0.89)
- Multi-Modal Grounded Planning and Efficient Replanning for Learning Embodied Agents with a Few Examples (AAAI 2025 · similarity 0.89)
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.88)
- Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning (NeurIPS 2025 · similarity 0.88)
- Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.