Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Multi-fingered Hand Grasps with Visuo-Tactile Fusion via Multi-Agent Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VT-MAGIC, a finger-level multi-agent DRL grasping scheme with a neuroscience-inspired self/cross-attention visuo-tactile fusion actor, evaluated in MuJoCo on 8 objects against single-agent DRL baselines.
Problem
Dexterous grasping with high-DoF multi-fingered hands is hard to control in the high-dimensional joint action space, and existing DRL grasping methods mostly use two-finger grippers or ignore tactile feedback when integrating vision and touch.
Approach
Treat the wrist, arm, and each of the five fingers as separate MADDPG agents with per-agent visuo-tactile actors (self-attention per modality then cross-attention fusion), with a single-agent approach phase to reach a pre-grasp pose and a single static reference grasp instead of human demonstrations.
Key results
- VT-MAGIC achieves 86% average grasp success over 50 trials per object, versus 75% for VT-SAPG, 74% for VT-SAPPO, 70% for SAPG, and 72% for SAPPO.
- Ablation: removing force feedback (VT-MAGIC-NF) lowers success to 82%, and replacing the fusion network with an MLP (VT-MAGIC-MLP) lowers it to 77%.
- Under shaking tests VT-MAGIC shows lower relative displacement on most objects and fewer drops than baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use finger-based multi-agent control with visuo-tactile fusion instead of the single-agent baselines | dexterous grasping · 8 objects | success rate: 70% -> 86%; 72% -> 86%; 74% -> 86%; 75% -> 86% | Helped |
| add tactile force feedback (ablation) | VT-MAGIC · 8 objects | success rate: 82% -> 86% | Helped |
| use the neuroscience-inspired visuo-tactile fusion network instead of a conventional MLP (ablation) | VT-MAGIC · 8 objects | success rate: 77% -> 86% | Helped |
| add tactile force feedback (ablation) | VT-MAGIC · 8 objects under shaking tests | maximum relative displacement | Helped |
| use the neuroscience-inspired visuo-tactile fusion network instead of a conventional MLP (ablation) | VT-MAGIC · 8 objects under shaking tests | maximum relative displacement | Helped |
Surprising findings
- The visuo-tactile baselines (VT-SAPG, VT-SAPPO) reach only 74-75%, and the conventional MLP fusion variant performs worse than the no-force ablation in success rate.
Limitations
- Evaluation is only in MuJoCo simulation; sim-to-real transfer is left as future work.
- Success rates are modest (86% best) and some objects (e.g., hammer, duck) show frequent falls under shaking.
- Reward functions and weighting coefficients are hand-designed and specific to each setup.
Experimental setup
Benchmarks: MuJoCo simulated grasping of 8 objects
Metrics: grasp success rate maximum relative displacement under shaking number of shakes before drop
Baselines: SAPG SAPPO VT-SAPG VT-SAPPO VT-MAGIC-NF VT-MAGIC-MLP
Builds on: MADDPG DAPG natural policy gradient PPO
Closest papers by meaning
- Enhancing Tactile-based Reinforcement Learning for Robotic Control (NeurIPS 2025 · similarity 0.89)
- Cross-Embodiment Dexterous Grasping with Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.88)
- ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.85)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.85)
- ForceFlow: Learning to Feel and Act via Contact-Driven Flow Matching (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.