Research map / 3D vision / Human pose estimation
3D Equivariant Visuomotor Policy Learning via Spherical Projection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Image-to-Sphere Policy (ISP), an SO(3)-equivariant diffusion policy that lifts eye-in-hand RGB features onto a sphere with gripper-orientation equivariance correction, outperforming Diffusion Policy and EquiDiff on MimicGen and real-world tasks.
Problem
Equivariant diffusion policies rely on multi-camera point clouds or fixed viewpoints, so they do not apply to monocular eye-in-hand RGB settings where SO(3) symmetry is hard to exploit and data efficiency matters.
Approach
Project 2D RGB encoder features onto a hemisphere via learnable orthographic projection, represent them as spherical Fourier coefficients, and rotate them by the gripper orientation (equivariance correction) so the observation encoding is SO(3)-equivariant; the diffusion U-Net is made equivariant by sampling over finite subgroups (C8, I60).
Key results
- ISP-SO(3) averages 65.2% success on 12 MimicGen tasks with 100 demos, +11.6 points over the best baseline.
- On four real-world UR5 tasks (20 trials each), ISP-SO(3) reaches 80%, 85%, 75%, 95% versus Diffusion Policy's 10%, 65%, 15%, 75%.
- Ablation shows removing spherical projection causes the largest drop (-9.2 points mean).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ISP-SO(3) instead of Diffusion Policy | MimicGen · MimicGen (100 demos) | mean success rate (%): 53.6 -> 65.2 | Helped |
| Use ISP-SO(3) instead of EquiDiff | MimicGen · MimicGen (200 demos) | mean success rate (%): 64.5 -> 75.0 | Helped |
| Use ISP-SO(3) instead of Diffusion Policy | Real-world manipulation · Box-Pipe | success rate: 10% vs. 80% | Helped |
| Use ISP-SO(3) instead of Diffusion Policy | Real-world manipulation · U-Pipe | success rate: 65% vs. 85% | Helped |
| Use ISP-SO(3) instead of Diffusion Policy | Real-world manipulation · 3D-Pipe | success rate: 15% vs. 75% | Helped |
| Use ISP-SO(3) instead of Diffusion Policy | Real-world manipulation · Grocery Bag | success rate: 75% vs. 95% | Helped |
| Add spherical projection and spherical convolutions (ablation) | ISP-SO(3) · MimicGen ablation tasks (100 demos) | mean success rate (%): 51.8 -> 61.0 (-9.2) | Helped |
| Add the equivariant image encoder (ablation) | ISP-SO(3) · MimicGen ablation tasks (100 demos) | mean success rate (%): 54.2 -> 61.0 (-6.8) | Helped |
| Add the equivariant temporal U-Net (ablation) | ISP-SO(3) · MimicGen ablation tasks (100 demos) | mean success rate (%): 54.3 -> 61.0 (-6.7) | Helped |
| Initialize the equivariant image encoder with ImageNet-1k-pretrained ResNet-18 (ablation) | ISP-SO(2) · MimicGen (100 demos) | mean success rate (%): 65.0 -> 72.1 (+7.1) | Helped |
| Use delta actions and remove rotation correction instead of absolute actions (ablation) | ISP-SO(2) · Square D2 (100 demos) | success rate: 32 -> 22 (-10) | Hurt |
| Use delta actions and remove rotation correction instead of absolute actions (ablation) | ISP-SO(2) · Nut Assembly D0 (100 demos) | success rate: 74 -> 57 (-17) | Hurt |
Surprising findings
- Delta (gripper-frame) actions without rotation correction underperform absolute actions, even for the standard Diffusion Policy.
Limitations
- Only a single wrist-mounted camera is used, lacking global scene context.
- Translational equivariance is not addressed, limiting generalization to object translations.
- Equivariant networks substantially increase training time.
Experimental setup
Datasets: MimicGen
Benchmarks: MimicGen
Metrics: success rate
Baselines: Diffusion Policy EquiDiff ACT
Builds on: Diffusion Policy EquiDiff escnn e3nn
Closest papers by meaning
- SE(3)-Equivariant Diffusion Policy in Spherical Fourier Space (ICML 2025 · similarity 0.91)
- A Practical Guide for Incorporating Symmetry in Diffusion Policy (NeurIPS 2025 · similarity 0.89)
- ET-SEED: EFFICIENT TRAJECTORY-LEVEL SE(3) EQUIVARIANT DIFFUSION POLICY (ICLR 2025 · similarity 0.88)
- Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning (NeurIPS 2024 · similarity 0.86)
- PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning (NeurIPS 2025 · similarity 0.86)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.