Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GauDP, which reconstructs a 3D Gaussian field from multi-view RGB with a fine-tuned NoPoSplat and dispatches per-view Gaussian features back to each agent's diffusion policy for multi-arm collaborative manipulation on RoboFactory.
Problem
Coordinating embodied multi-agent manipulation requires balancing fine-grained local control with global scene awareness; image-based diffusion policies lack 3D structure while point-cloud methods need extra sensing.
Approach
Reconstruct a globally consistent 3D Gaussian field from decentralized RGB views with a feed-forward pose-free model, then pixel-align and dispatch each agent's own-view Gaussians back into its image features via a lightweight convolutional fusion, using RGB-only input.
Key results
- GauDP achieves 19.67% average success over 6 RoboFactory tasks, versus 14.33% for the best point-cloud baseline DP3(XYZ+RGB) and 5.00% for image-based DP.
- On Lift Barrier (2 arms), GauDP reaches 72% success vs. 31% for DP3(XYZ+RGB).
- Fine-tuned Gaussian reconstruction improves PSNR from 17.918 to 23.424 over the pretrained model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GauDP's Gaussian-image representation and policy | multi-agent manipulation · RoboFactory (6 tasks) | average success rate: 14.33% -> 19.67% | Helped |
| Fine-tune Noposplat on manipulation observations | 3D Gaussian reconstruction · 3D Gaussian reconstruction | PSNR: 17.918 -> 23.424 | Helped |
| Fine-tune Noposplat on manipulation observations | 3D Gaussian reconstruction · 3D Gaussian reconstruction | LPIPS: 0.492 -> 0.148 | Helped |
| Use GauDP instead of DP | real-robot multi-agent tasks · Card Box Stacking | successful executions: 11/30 -> 17/30 | Helped |
| Use GauDP instead of DP | real-robot multi-agent tasks · Card Box Handover | successful executions: 14/30 -> 19/30 | Helped |
| Use GauDP instead of DP | real-robot multi-agent tasks · Grab Roller | successful executions: 22/30 -> 27/30 | Helped |
| Add local camera-coordinate Gaussian parameterization instead of unified world coordinates (ablation) | GauDP · RoboFactory (6 tasks) | average success rate: 10.83% -> 19.67% | Helped |
| Add pixel-level prefusion instead of coarse feature-level fusion (ablation) | GauDP · RoboFactory (6 tasks) | average success rate: 1.17% -> 19.67% | Helped |
| Add image input (ablation) | GauDP · RoboFactory (6 tasks) | average success rate: 11.17% -> 19.67% | Helped |
| Add Gaussian representation (ablation) | GauDP · RoboFactory (6 tasks) | average success rate: 5.00% -> 19.67% | Helped |
| Use a shared global policy instead of independent local policies (ablation) | GauDP · RoboFactory (6 tasks) | average success rate: 5.33% -> 19.67% | Helped |
| Use GauDP rather than parameter-matched LargeDP | multi-agent manipulation · RoboFactory (6 tasks) | average success rate: 17.50% -> 19.67% | Helped |
Surprising findings
- GauDP's average success (19.67%) is low in absolute terms, with several tasks at 0-3%, including Stack Cube (3 arms) at 0%.
- A scaled-up LargeDP with matched parameter count reaches 17.50% average, close to GauDP's 19.67%, and GauDP is worse on Stack Cube (2 arms) than DP.
Limitations
- Limited number and diversity of embodied tasks hinders training a generalizable Gaussian reconstruction network.
- Framework does not incorporate multimodal inputs or semantic features.
- Absolute success rates remain low on several multi-arm tasks.
Experimental setup
Datasets: RoboFactory
Benchmarks: RoboFactory
Metrics: success rate PSNR SSIM LPIPS
Baselines: Diffusion Policy 2D Dense Policy 3D Diffusion Policy (DP3) 3D Dense Policy LargeDP
Builds on: Diffusion Policy 3D Diffusion Policy NoPoSplat 3D Gaussian Splatting RoboFactory
Closest papers by meaning
- MAC-Ego3D: Multi-Agent Gaussian Consensus for Real-Time Collaborative Ego-Motion and Photorealistic 3D Reconstruction (CVPR 2025 · similarity 0.88)
- COMBO: Compositional World Models for Embodied Multi-Agent Cooperation (ICLR 2025 · similarity 0.86)
- MAGiC-SLAM: Multi-Agent Gaussian Globally Consistent SLAM (CVPR 2025 · similarity 0.86)
- An Extensible Framework for Open Heterogeneous Collaborative Perception (ICLR 2024 · similarity 0.86)
- VIKI‑R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Act to See, See to Act: Diffusion-Driven Perception-Action Interplay for Adaptive Policies (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.