Research map / Large language models / Language, safety and interpretability
Learning with Language-Guided State Abstractions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Language-Guided Abstraction (LGA), where GPT-4 turns a language task description into a state abstraction (object/color feature mask) that an imitation policy is trained over, matching human-designed abstractions with far less user time.
Problem
Learning generalizable imitation policies from few demonstrations in high-dimensional, distractor-rich observation spaces requires task-relevant state abstractions, which are costly to hand-design; existing LM-for-robotics methods rely on skill libraries or environment interaction.
Approach
Use an LM to select task-relevant features from a textualized scene feature set given a language task description, then render these as a binary goal mask over the observation and train an imitation policy on the abstracted state, so the policy never sees the language input.
Key results
- LGA outperforms GCBC-DART and other baselines on task performance vs. number of demonstrations (10 to 50) across 13 tasks, and matches human-specified abstractions.
- LGA reduces user specification time significantly compared to manual Human feature specification (p < 0.001, paired t-test).
- On state covariate shift tests, LGA variants outperform GCBC+DART; LGA also resolves unseen linguistic commands zero-shot in multi-task settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LGA-generated task-relevant state abstractions instead of raw-observation goal-conditioned behavior cloning | LGA · VIMA Q1, 13 scenarios | success rate vs. number of demonstrations | Helped |
| Use LM-assisted feature specification rather than manually specifying features | LGA and LGA-HILL · VIMA Q1, 13 scenarios | user specification time: p < 0.001 | Helped |
| Add human refinement to the LM-proposed abstraction | LGA · VIMA Q1, 13 scenarios | success rate | No clear effect |
| Use state abstractions, with or without the raw observation, rather than GCBC-DART | LGA · VIMA Q2 observational shifts | success rate | Helped |
| Replace the state abstraction with only a language embedding of relevant features (ablation) | LGA · VIMA Q2 observational shifts | success rate | Hurt |
| Use LGA state abstractions to resolve unseen linguistic commands | LGA · VIMA Q3 multi-task linguistic ambiguity | success rate | Helped |
| Replace the state abstraction with only a language embedding of relevant features (ablation) | LGA · VIMA Q3 multi-task linguistic ambiguity | success rate | Hurt |
| Use LGA-generated abstractions for real-world mobile manipulation | LGA · Spot bring fruit and throw away drink | task completion | Helped |
Surprising findings
- Ablation LGA-L (conditioning only on language abstraction) underperforms state-abstraction variants, showing grounding in observations matters.
Limitations
- Assumes task-relevant features are visually segmentable and expressible in language; less applicable to relative position or trajectory-dependent features.
- Real-world Spot experiments are limited to a single demonstration per task, with failures attributed to captioning errors.
- Evaluated only in the simulated VIMA environment with a fixed feature space of 29 objects and 81 textures for the main experiments.
Experimental setup
Datasets: VIMA simulated manipulation tasks (generated) Oracle demonstrations generated in simulation
Benchmarks: VIMA-based pick-and-place, rotate, sweep-while-avoid tasks Spot mobile manipulation tasks
Metrics: task success rate user specification time (s) success under state covariate shift multi-task success on seen and unseen utterances
Baselines: GCBC-DART GCBC-SEG Human (manual feature specification) LGA-L
Builds on: VIMA DART Sentence-BERT Segment Anything
Closest papers by meaning
- Grounding Language Plans in Demonstrations Through Counterfactual Perturbations (ICLR 2024 · similarity 0.89)
- Policy Improvement using Language Feedback Models (NeurIPS 2024 · similarity 0.89)
- Learning Planning Abstractions from Language (ICLR 2024 · similarity 0.88)
- VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning (ICLR 2025 · similarity 0.88)
- Instant Policy: In-Context Imitation Learning via Graph Diffusion (ICLR 2025 · similarity 0.88)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.