Research map / Vision-language and multimodal / Robot action and navigation
Memory and subgoal-based manipulation
Improves robot manipulation with memory, behavioral phasing, subgoal images and skill decomposition.
103 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2020–2026) · an approach in Robot action and navigation, keyword cluster “vla · language action · robotic”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
14 papers in 2023–24 and 86 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 18.4% to 17.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2020 · Visual Grounding of Learned Physical Models (ICML 2020 · 22 citations)
- 2021 · Learning Intuitive Physics with Multimodal Generative Models (AAAI 2021 · 5 citations)
- 2022 · ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment (NeurIPS 2022 · 9 citations)
- 2023 · ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation (NeurIPS 2023 · 10 citations)
- 2024 · ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · 82 citations)
- 2025 · Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection (CVPR 2025 · 13 citations)
- 2026 · DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping (AAAI 2026 · 9 citations)
Earliest papers here
- Visual Grounding of Learned Physical Models (ICML 2020 · 22 citations)
- Learning Intuitive Physics with Multimodal Generative Models (AAAI 2021 · 5 citations)
- ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment (NeurIPS 2022 · 9 citations)
What has been tried
38 papers here report 397 measured changes (a component added, removed or swapped, compared with a baseline): helped 75%, no clear effect 7%, hurt 18%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add object category identification to fine-tuning | Novel instances in train categories · success rate | 0.41 -> 0.44; +3% | Helped | ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · 82 citations) |
| add cross-object correspondence learning | Folding (tops, dresses, trousers) · success rate | 75.3 -> 83.6; 70.8 -> 86.9; 77.6 -> 81.6 | Helped | UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence (CVPR 2024 · 20 citations) |
| add Levels 1–4 simulator data augmentation instead of using Level 1 only | Real-world settings · success rate | 20% -> 35%; 5% -> 30%; 0% -> 30%; 0% -> 40% | Helped | CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation (CVPR 2024 · 15 citations) |
| Add constraint elements to CaM | Reasoning Pick & Place: grasp animals by distance to fruits · success rate (%) | 20% -> 60% | Helped | Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection (CVPR 2025 · 13 citations) |
| add channel concatenation and grounded perceiver | RoboCasa Appearance pick-and-place (ablation subset) · success rate | 12 -> 32 | Helped | RoboGround: Robotic Manipulation with Grounded Vision-Language Priors (CVPR 2025 · 11 citations) |
| Insert an adapter before the first backbone block | Adroit · average success rate | 74.0 -> 79.6 (+5.6) | Helped | Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation (CVPR 2025 · 7 citations) |
| add coarse semantic alignment pre-training | Visual rearrange · FID | 23.1 -> 11.3 | Helped | Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts (CVPR 2024 · 7 citations) |
| replace diffusion modeling with deterministic regression | Wash Cup, unseen cup 2 · success rate (%) | 50 -> 12.5 | Hurt | RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (ICLR 2025 · 5 citations) |
| co-train the motion prediction model on expert demonstrations and correction data | RoboMimic (9 tasks) · success rate | 46.9% -> 49.6% | Helped | Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correction (CVPR 2025 · 4 citations) |
| Add semantic orientation to the scene graph | Open6DOR V1 Rotation Track · success rate | 13.0 -> 57.0 | Helped | SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation (NeurIPS 2025 · 4 citations) |
| Increase Aha instruction-tuning data size | ManiSkill-Fail · performance across four evaluation metrics | average quadratic fit gradient of 0.0022 | Helped | AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR 2025 · 4 citations) |
| Add large-scale video pre-training and video prediction | CALVIN ABCD→D · average length | 3.33 -> 4.21 | Helped | Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 36 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- LIBERO · 5 of 36 papers
- DROID · 5 of 36 papers
- BridgeData V2 · 4 of 36 papers
- RLBench · 3 of 36 papers
- LIBERO-90 · 3 of 36 papers
- LIBERO-LONG · 3 of 36 papers
- Open X-Embodiment · 3 of 36 papers
- RH20T · 2 of 36 papers
Baselines
- OpenVLA · 7 of 36 papers
- ACT · 4 of 36 papers
- pi0 · 4 of 36 papers
- R3M · 3 of 36 papers
- Diffusion Policy · 3 of 36 papers
- 3D Diffuser Actor · 3 of 36 papers
- GPT-4o · 3 of 36 papers
- RVT · 2 of 36 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 100 papers; from OpenAlex and the papers' affiliations).
- Peking University · 7 papers
- Shanghai Jiao Tong University · 6 papers
- Tsinghua University · 6 papers
- Shanghai AI Laboratory · 5 papers
- Sun Yat-sen University · 4 papers
- Beihang University · 3 papers
- Beijing Academy of Artificial Intelligence · 3 papers
- Zhejiang University · 3 papers
- NVIDIA · 3 papers
- Tencent · 3 papers
Use it
Download BibTeX of 24 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Vision-language-action foundation models (264 papers)
- Embodied multimodal LLM agents (70 papers)
- Multimodal prompt-driven manipulation (50 papers)
Elsewhere in the topic
- Pretraining and data augmentation for VLN (97 papers)
- Reasoning-enhanced vision-language navigation (46 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).