Research map / Vision-language and multimodal / Robot action and navigation
AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AC-DiT, an end-to-end diffusion transformer for language-conditioned mobile manipulation that first predicts base motion to condition whole-body actions and adaptively weights 2D image and 3D point-cloud features.
Problem
End-to-end mobile manipulation struggles to coordinate the mobile base with the manipulator, leading to error accumulation, and existing methods use a single visual modality (2D or 3D) across all stages of a task.
Approach
A mobility-to-body conditioning mechanism pretrains a lightweight DiT mobility head on base actions and feeds its latent features as a prior for whole-body action prediction, combined with a perception-aware adaptation that weights 2D multi-view and 3D point-cloud tokens by cosine similarity to the language feature.
Key results
- On ManiSkill-HAB, AC-DiT reaches a mean success rate of 55.6% vs 42.9% for RDT and 28.8% for DP.
- On RoboTwin dual-arm tasks, AC-DiT achieves 90.1% mean success vs 73.3% for 3DP and 70.0% for RDT.
- Ablation on the real-world-style setup: adding multimodal fusion, mobility-to-body conditioning and perception-aware adaptation raises mean success from 37.5% to 49.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AC-DiT instead of RDT | ManiSkill-Hab Set-Table · ManiSkill-Hab Set-Table | mean success rate: 42.9 -> 55.6 | Helped |
| Use AC-DiT instead of 3DP | RoboTwin · RoboTwin | mean success rate: 73.3 -> 90.1 | Helped |
| Add 3D visual input to 2D-only input (ablation) | AC-DiT · ManiSkill-Hab ablation | mean success rate: 37.5% -> 44.8% | Helped |
| Add Mobility-to-Body Conditioning (MBC) to 2D + 3D (ablation) | AC-DiT · ManiSkill-Hab ablation | mean success rate: 44.8% -> 47.0% | Helped |
| Add Perception-Aware Multimodal Adaptation (PMA) to MBC (ablation) | AC-DiT · ManiSkill-Hab ablation | mean success rate: 47.0% -> 49.0% | Helped |
| Use alternating rather than non-alternating cross-attention condition injection (ablation) | AC-DiT · Condition injection ablation | mean: 44.8 -> 43.7 | Hurt |
| Condition on latent mobility features rather than latent manipulator features (ablation) | AC-DiT · Conditioning relationship ablation | mean: 47.3 -> 49.5 | Helped |
Surprising findings
- Mobility-to-body conditioning outperforms conditioning on latent manipulator (arm joint) features (49.5 vs 47.3 mean in the appendix ablation).
Limitations
- As an imitation learning method, performance depends on demonstration quantity and quality and inherits suboptimal or biased demonstrations.
- Locomotion errors from velocity-controlled base can accumulate and lead to unrecoverable states.
- Poor locomotion termination leads to manipulation failures when the base stops outside the training distribution.
Experimental setup
Datasets: ManiSkill-HAB demonstrations (RL-generated) RoboTwin demonstrations (motion-planning generated) real-world Agilex Cobot Magic demonstrations
Benchmarks: ManiSkill-HAB (Set-Table, 7 tasks) RoboTwin (6 dual-arm tasks) real-world Agilex Cobot Magic (4 long-horizon tasks)
Metrics: success rate (mean ± std over 3 runs of 100 episodes)
Baselines: ACT Diffusion Policy 3D Diffusion Policy RDT pi0
Builds on: Diffusion Policy 3D Diffusion Policy Robotics Diffusion Transformer Lift3D SigLIP
Closest papers by meaning
- MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulation (CVPR 2025 · similarity 0.89)
- SPIN: Simultaneous Perception Interaction and Navigation (CVPR 2024 · similarity 0.88)
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (ICLR 2025 · similarity 0.88)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.88)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.87)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.