Research map / Vision-language and multimodal / Robot action and navigation
GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GravMAD, a sub-goal-driven language-conditioned action diffusion policy that conditions 3D diffuser actor on grounded spatial value maps (GravMaps) built from sub-goals discovered in demonstrations (training) or predicted by GPT-4o/Semantic-SAM pipelines (inference).
Problem
Imitation-learned 3D manipulation policies overfit to training tasks and fail on novel instructions/scenes, while foundation-model pipelines lack precise, task-specific 3D grounding; the paper seeks precise and generalizable instruction-following 3D manipulation.
Approach
Discover sub-goal keyposes from demonstrations (gripper-state and touch-force changes) for training and use foundation models (SoM Detector, LLM Planner, code Composer) to predict sub-goals at inference; convert sub-goals into voxel cost and gripper-openness maps (GravMaps), encode them as tokens with a DP3 encoder, and condition a 3D Diffuser Actor-style denoiser on them with auxiliary sub-goal l
Key results
- GravMAD achieves 62.92% average success on 8 novel RLBench tasks, +28.63 over the best baseline (VoxPoser).
- GravMAD (Manual) reaches 69.17% average success on 12 base tasks, +13.36 over 3D Diffuser Actor; GravMAD (VLM) reaches 56.72%.
- Ablations show removing GravMap or auxiliary/contrastive losses degrades performance; replacing DP3 with PointNet or PointNet++ also lowers results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GravMAD with VLM-generated GravMaps instead of the strongest baseline | 3D Diffuser Actor · RLBench novel tasks (8-task average) | average success rate: +28.63 | Helped |
| Use GravMAD with manually provided context instead of the strongest baseline | 3D Diffuser Actor · RLBench base tasks (12-task average) | average success rate: +13.36 | Helped |
| Use GravMAD with VLM-generated context instead of 3D Diffuser Actor | 3D Diffuser Actor · RLBench base tasks (12-task average) | average success rate: +0.91 | Helped |
| Replace GravMaps with sub-goal position and openness values (ablation) | GravMAD · RLBench base and novel tasks | average success rate | Hurt |
| Remove the gripper map from GravMaps (ablation) | GravMAD · RLBench base and novel tasks | average success rate | Hurt |
| Remove the cost map from GravMaps (ablation) | GravMAD · RLBench base and novel tasks | average success rate | Hurt |
| Remove the contrastive learning loss (ablation) | GravMAD · RLBench base and novel tasks | average success rate | Hurt |
| Remove the auxiliary position and openness losses (ablation) | GravMAD · RLBench base and novel tasks | average success rate | Hurt |
| Condition rotation actions on GravMap tokens (ablation) | GravMAD · RLBench base and novel tasks | average success rate | Hurt |
| Replace the DP3 encoder with PointNet (ablation) | GravMAD · RLBench base and novel tasks | average success rate | Hurt |
| Replace the DP3 encoder with PointNet++ (ablation) | GravMAD · RLBench base and novel tasks | average success rate | Hurt |
Surprising findings
- GravMAD (VLM) underperforms 3D Diffuser Actor on open drawer, put in drawer and place wine due to detector coarseness.
- Conditioning rotation actions on GravMap tokens reduces performance.
Limitations
- Effectiveness depends heavily on prompt engineering.
- VLM detectors have limited detection capability and are sensitive to camera viewpoint changes.
- Sub-goal keypose discovery cannot handle tool-use tasks.
Experimental setup
Datasets: RLBench demonstrations (12 base tasks, 20 demos each)
Benchmarks: RLBench
Metrics: average success rate average rank
Baselines: VoxPoser Act3D 3D Diffuser Actor
Builds on: 3D Diffuser Actor VoxPoser Set-of-Mark DP3 keypose discovery
Closest papers by meaning
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.90)
- Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts (CVPR 2024 · similarity 0.89)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.89)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.89)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.89)
- Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.