Research map / Video / Action recognition
GraphMimic: Graph-to-Graphs Generative Modeling from Videos for Policy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a graph-to-graphs generative model on action-free videos to predict future object/interaction-point graphs, then trains a graph-conditioned policy on few action-labeled demonstrations.
Problem
Learning robot manipulation policies requires costly action-labeled demonstrations; existing video pre-training methods model pixels and neglect object structure, spatial relations, and cross-embodiment transfer.
Approach
Abstracts video frames into object vertices and visual action vertices (fingertips or end-effector interaction points), builds graphs, and pre-trains a hierarchical, property-aware graph generator to predict future graphs; the generated graphs condition a policy trained on few action-labeled demos.
Key results
- On LIBERO with 10 action-labeled demos per task, GraphMimic reaches 0.88 on Spatial, 0.89 on Object, 0.87 on Goal, 0.56 on Long, and 0.67 on LIBERO-90, outperforming ATM and UniPi.
- Using only 2 action-labeled demos per task (20%), GraphMimic still outperforms ATM and UniPi trained with 10 demos in simulation.
- Real-world overall success rate is 0.76 versus 0.53 for ATM; human-to-robot overall success is 0.64 versus 0.27 for ATM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GraphMimic instead of ATM with 10 action-labeled demos | Policy learning on LIBERO · LIBERO suites | success rate: 0.68 ± 0.02 -> 0.88 ± 0.02 (Libero-Spatial); 0.68 ± 0.06 -> 0.89 ± 0.02 (Libero-Object); 0.77 ± 0.01 -> 0.87 ± 0.03 (Lib | Helped |
| Use GraphMimic with 2 action-labeled demos instead of ATM with 10 | Real-world manipulation policy · Real-world manipulation | success rate: 0.53 -> 0.63 | Helped |
| Use GraphMimic instead of ATM with 10 action-labeled demos | Robot-to-robot transfer policy · Robot-to-robot transfer, Libero-Spatial | success rate: 0.45 ± 0.04 -> 0.81 ± 0.04 | Helped |
| Use GraphMimic instead of ATM with 10 action-labeled demos | Human-to-robot transfer policy · Human-to-robot transfer | success rate: 0.27 -> 0.64 | Helped |
| Use task-related graph vertices rather than grid-sampled vertices or point-based flow (ablation) | GraphMimic · Robot-to-robot transfer, Libero-Spatial and Libero-Goal | success rate: 0.51 -> 0.81 (Libero-Spatial); 0.53 -> 0.81 (Libero-Goal) | Helped |
| Add visual action vertices represented by interaction points (ablation) | GraphMimic · Robot-to-robot transfer, Libero-Spatial | success rate: 0.41 -> 0.81 | Helped |
| Use interaction points instead of end-effector vertices (ablation) | GraphMimic · Robot-to-robot transfer, Libero-Spatial | success rate: 0.56 -> 0.81 | Helped |
| Represent task-related objects rather than manipulated objects only or all objects (ablation) | GraphMimic · Robot-to-robot transfer, Libero-Spatial | success rate: 0.71 -> 0.81 (manipulated objects); 0.55 -> 0.81 (all objects) | Helped |
| Use hierarchical graph modeling instead of local-only or global-only connections (ablation) | GraphMimic · Robot-to-robot transfer, Libero-Spatial | success rate: 0.73 -> 0.81 (local connection); 0.75 -> 0.81 (global connection) | Helped |
| Add object property estimation (ablation) | GraphMimic · Real-world manipulation and human-to-robot transfer | success rate: 0.66 -> 0.76 (real-world); 0.57 -> 0.64 (human-to-robot) | Helped |
| Add an LSTM to object property estimation (ablation) | GraphMimic · Real-world manipulation and human-to-robot transfer | success rate: 0.73 -> 0.76 (real-world); 0.60 -> 0.64 (human-to-robot) | Helped |
Surprising findings
- Graph representation with 2 action-labeled demos outperforms the BC baseline trained on the full action dataset in simulation.
Limitations
- Current tracking models struggle with low-texture objects during large motions, compromising training data quality.
- Relies on VLM and segmentation pipelines to identify task-relevant objects.
Experimental setup
Datasets: LIBERO self-collected robot videos human videos
Benchmarks: LIBERO-Spatial LIBERO-Object LIBERO-Goal LIBERO-Long LIBERO-90 real-world manipulation tasks
Metrics: success rate
Baselines: BC R3M UniPi ATM Im2Flow2Act Diffusion policy
Builds on: ATM UniPi R3M CoTracker Grounded SAM
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.89)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.88)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Learning to Act from Actionless Videos through Dense Correspondences (ICLR 2024 · similarity 0.88)
- Action Images: End-to-End Policy Learning via Multiview Video Generation (NeurIPS 2026 · similarity 0.87)
- SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.