Research map / Multi-agent systems and games / Human-AI and goal-conditioned
SPIN: Simultaneous Perception Interaction and Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a single RL policy on a Hello Robot Stretch to jointly control base, arm and actuated camera for active-perception mobile manipulation in clutter, using privileged scandots in RL then distilling to ego-depth via DAgger.
Problem
Mobile manipulation in cluttered, dynamic environments requires coupling whole-body control with active perception, since the robot's limited camera field of view determines what it knows and how it can move; classical map-plan-execute pipelines fail under noise and dynamic obstacles.
Approach
Cast whole-body navigation and manipulation with active camera control as a single reactive end-to-end policy: train with RL on privileged visible scandots (either jointly as CVO or decoupled as DVO), then distill to a depth-image student via supervised learning/DAgger.
Key results
- In simulation, SPIN (CVO) reaches 0.93 success on hard reach and 0.97 on hard pick in Scenario 1, versus 0.53 and 0.20 for FixCam on medium and hard reach, showing active vision is needed.
- SPIN achieves about 33% higher success than the NoPointNet baseline and about 68% higher than FixCam in simulation, per the paper.
- In real-world dynamic-obstacle tests, the classical mapping baseline has near-zero success while SPIN succeeds, and SPIN succeeds 20-40% more than the classical baseline overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPIN instead of classical mapping and planning | Mobile navigation · Real-world static and dynamic obstacle scenarios | success rate: 20-40% more | Helped |
| Add active camera movement instead of fixing the camera forward (ablation) | SPIN · Simulation cluttered navigation | success rate: ≈68% | Helped |
| Use PointNet for scandot encoding instead of concatenation and an MLP (ablation) | SPIN · Simulation cluttered navigation | success rate: ≈33% | Helped |
| Use decoupled visuomotor optimization instead of coupled visuomotor optimization (ablation) | SPIN · Simulation benchmarks | success rate | No clear effect |
Surprising findings
- Coupled (CVO) and decoupled (DVO) visuomotor optimization perform similarly despite the expected advantage of joint optimization.
- The policy avoids dynamic obstacles it never saw during simulation training, emerging as a by-product of diverse procedural environments.
Limitations
- Relies on stereo-matched depth rather than RGB, so it can bump into glass or shiny surfaces.
- CVO requires billions of samples, which may not always be feasible in practice.
- Real-world evaluation is limited to two environments and compares only against a classical mapping baseline.
Experimental setup
Datasets: procedurally generated simulation corridors with obstacles (IsaacGymEnvs)
Benchmarks: 6 simulation obstacle-course benchmarks (easy, medium, hard; 2 scenarios each) 2 real-world environments
Metrics: success rate average episode duration average number of collisions
Baselines: FixCam NoPointNet Mapping Classical (gmapping + move_base)
Builds on: PPO DAgger PointNet
Closest papers by meaning
- AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation (NeurIPS 2025 · similarity 0.88)
- Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation (AAAI 2025 · similarity 0.86)
- ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.85)
- MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulation (CVPR 2025 · similarity 0.85)
- OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis (NeurIPS 2025 · similarity 0.85)
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.