Research map / Large language models / Agents, code and math
GNN-Transformer Task Planning Enhanced with Semantic-Driven Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GTTP, a GNN-Transformer task planner over text-embedded scene graphs, trained on a 14k-instance dataset generated by LLM-verified semantic augmentation of 45 seed tasks in VirtualHome, and deployed on a real mobile manipulator.
Problem
Instruction-driven high-level robot task planning in household environments suffers from weak physical grounding, prompt dependence, oversized LLM planners, high planning cost, and scarce task-planning training data.
Approach
A small GNN over Sentence-BERT-embedded scene graphs plus a transformer over state history predicts the next action and target object, with feasibility-checked subgoal selection; training data is expanded by semantic object substitution within rooms and LLM (GPT-4o) plan verification and instruction generation.
Key results
- GTTP achieves 76.1% GCR and 62.9% SR in seen VirtualHome environments versus 36.3% SR for the best LLM baseline (GPT-4-T zero-shot), and 57.7% SR in unseen environments.
- Ablation shows removing the progress context fusion module drops seen SR from 62.9% to 17.6%.
- On a real Husky + Franka mobile manipulator with 7 tasks, GTTP achieves 100% GCR and SR on goal-instruction and step-by-step formats for most tasks, and 83.3% success over 18 trials with GPT-4o-built scene graphs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GTTP instead of the strongest reported baseline, ZERO-SHOT PLANNER (GPT-3.5-TI) | VirtualHome task planning · Seen environments | SR: 44.8± 0.2 vs 62.9 | Helped |
| use GTTP instead of the strongest reported baseline, ZERO-SHOT PLANNER (GPT-3.5-TI) | VirtualHome task planning · Unseen environments | SR: 46.0±1.0 vs 57.7 | Helped |
| add the progress context fusion module (ablation) | GTTP · VirtualHome seen environments | SR: 17.6 -> 62.9 | Helped |
| add the progress context fusion module (ablation) | GTTP · VirtualHome unseen environments | SR: 22.9 -> 57.7 | Helped |
| add the progress context fusion module (ablation) | GTTP · Real-world tasks | SR: 7.1 -> 78.6 | Helped |
| add global update function ρu to the GNN layers (ablation) | GTTP · VirtualHome seen environments | SR: 51.5 -> 62.9 | Helped |
| add global update function ρu to the GNN layers (ablation) | GTTP · VirtualHome unseen environments | SR: 53.1 -> 57.7 | Helped |
| add global update function ρu to the GNN layers (ablation) | GTTP · Real-world tasks | SR: 57.1 -> 78.6 | Helped |
| add node dropout during training (ablation) | GTTP · VirtualHome seen environments | SR: 60.3 -> 62.9 | Helped |
| add node dropout during training (ablation) | GTTP · VirtualHome unseen environments | SR: 60.6 -> 57.7 | Hurt |
| add node dropout during training (ablation) | GTTP · Real-world tasks | SR: 14.3 -> 78.6 | Helped |
Surprising findings
- GPT-4o zero-shot planner scores lower than GPT-4-Turbo and GPT-3.5-Turbo-Instruct in success rate in the seen environment (6.3% SR).
Limitations
- Performance drops substantially for long task plans (SR 44.7% on long plans).
- Real-world experiments use small custom scenes and manually constructed scene graphs; failures on objects absent from training (table lamp).
- Plan verification and augmentation are limited in diversity of task plans.
Experimental setup
Datasets: VirtualHome seed tasks from ProgPrompt (Singh et al. 2023) hand-crafted VirtualHome tasks
Benchmarks: VirtualHome seen environment test VirtualHome unseen environment test real-world mobile manipulator tasks
Metrics: Executability Goal Conditions Recall Success Rate LLM API cost Plan SR
Baselines: Zero-Shot Planner ProgPrompt GoalNet GRID
Builds on: Graph Networks (Battaglia et al. 2018) Transformer Sentence-BERT chain-of-thought prompting VirtualHome
Closest papers by meaning
- Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.88)
- OntoPlan: An Ontology-Grounded Scene Representation and Agentic Framework for Scalable Robot Task Planning (NeurIPS 2026 · similarity 0.88)
- GOALNET: Interleaving Neural Goal Predicate Inference with Classical Planning for Generalization in Robot Instruction Following (AAAI 2024 · similarity 0.88)
- R2C: Mapping Room to Chessboard to Unlock LLM As Low-Level Action Planner (CVPR 2025 · similarity 0.88)
- Video Language Planning (ICLR 2024 · similarity 0.88)
- Language Guided Skill Discovery (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.