Research map / Large language models / Agents, code and math
Grounding Language Plans in Demonstrations Through Counterfactual Perturbations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses LLMs to propose a mode-family structure and feasibility matrix for a task, then learns mode classifiers from counterfactually perturbed demonstrations with only sparse task-success labels, and builds mode-based imitation policies and LLM replanning.
Problem
Grounding LLM common-sense task knowledge into low-level physical robot trajectories for multi-step manipulation, without dense mode annotations or pre-built primitive skills.
Approach
Perturb a few successful human demonstrations (end-effector and gripper perturbations) to create both successful and failing trajectories, use LLM-generated mode sequences and a feasibility matrix as structure, and train a mode classifier end-to-end with a differentiable task-success loss (plus a dynamics loss for underactuated systems).
Key results
- On 2D navigation, GLiDE+Planning achieves 0.991 success (0.974 with perturbations) in the 5-mode environment versus 0.810 (0.596) for BC.
- Mode classification accuracy in the 2D polygon environment reaches 0.975-0.989, while removing failed counterfactuals drops it to about 0.67-0.80.
- On RoboSuite Square, GLiDE+BC under perturbation scores 0.25 versus 0.38 for BC, so the claimed robustness benefit is not uniform across tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GLiDE+Planning instead of behavior cloning | 2D navigation · 2D navigation 3-, 4-, and 5-Mode, with and without perturbations | success rate: 0.967 -> 0.996; 0.908 -> 0.996; 0.814 -> 0.987; 0.614 -> 0.966; 0.810 -> 0.991; 0.596 -> 0.974 | Helped |
| Use GLiDE+BC instead of behavior cloning | 2D navigation · 2D navigation 4-Mode and 5-Mode, with and without perturbations | success rate: 0.814 -> 0.892; 0.614 -> 0.753; 0.810 -> 0.893; 0.596 -> 0.753 | Helped |
| Use mode-conditioned imitation instead of mode-agnostic behavior cloning | RoboSuite · RoboSuite Can, Lift, and Square with perturbations | success rate: 0.20 -> 0.40; 0.18 -> 0.39; 0.03 -> 0.15 | Mixed |
| Use mode-conditioned imitation instead of behavior cloning | RoboSuite · RoboSuite Can, Lift, and Square without perturbations | success rate: 0.93 -> 0.85; 0.99 -> 0.99; 0.38 -> 0.25 | Mixed |
| Add the dynamics loss (ablation) | GLiDE · RoboSuite Can, Lift, and Square | mode classification accuracy (overlap): 0.67 -> 0.83; 0.75 -> 0.83; 0.46 -> 0.67 | Helped |
| Add the task prediction loss (ablation) | GLiDE · RoboSuite Can, Lift, and Square | mode classification accuracy (overlap): 0.67 -> 0.83; 0.68 -> 0.83; 0.56 -> 0.67 | Helped |
| Add LLM-based feature selection (ablation) | GLiDE · RoboSuite Can, Lift, and Square | mode classification accuracy (overlap): 0.55 -> 0.83; 0.70 -> 0.83; 0.57 -> 0.67 | Helped |
| Use both successful and failed perturbation trajectories instead of retaining only successful trajectories (ablation) | GLiDE · 2D navigation 3-, 4-, and 5-Mode | mode classification accuracy: 0.701 -> 0.989; 0.669 -> 0.989; 0.798 -> 0.975 | Helped |
Surprising findings
- On RoboSuite Square with perturbations, GLiDE+BC (0.15) is not better than BC (0.03) in the no-perturbation-free case; in the unperturbed setting GLiDE+BC (0.25) underperforms BC (0.38), indicating mode-based BC does not uniformly help.
- Setting all negative feasibility entries to -1 can cause gradient descent to get stuck, so the paper uses a graded negative-shortest-path matrix.
Limitations
- Requires many trial-and-error rollouts and a resettable environment to obtain task-success labels.
- Prompting the LLM for a good state representation requires skill.
- Mode-based BC does not uniformly outperform the baseline on all RoboSuite tasks (e.g., Square).
Experimental setup
Datasets: 2D navigation polygon environment RoboSuite Can RoboSuite Lift RoboSuite Square real Franka 2D navigation trajectories marble scooping demonstrations
Benchmarks: 2D navigation polygon environment RoboSuite Can RoboSuite Lift RoboSuite Square
Metrics: success rate mode classification accuracy (overlap with ground truth)
Baselines: Behavior Cloning (BC) kmeans++ trajectory segmentation GLiDE without fail data GLiDE without dynamics loss GLiDE without prediction loss GLiDE without feature selection
Builds on: Behavior cloning RRT Segment Anything Model
Closest papers by meaning
- Learning with Language-Guided State Abstractions (ICLR 2024 · similarity 0.89)
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.88)
- Grounding Multimodal Large Language Models in Actions (NeurIPS 2024 · similarity 0.88)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.87)
- RoboGround: Robotic Manipulation with Grounded Vision-Language Priors (CVPR 2025 · similarity 0.87)
- Multi-Modal Grounded Planning and Efficient Replanning for Learning Embodied Agents with a Few Examples (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.