Research map / Point clouds / Point cloud classification
PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PointMapPolicy, which feeds structured XYZ point maps (same grid as the RGB image) into an EDM diffusion policy with an xLSTM backbone, and fuses them with RGB tokens for imitation learning.
Problem
Point cloud policies must downsample (FPS) or use irregular architectures, losing fine geometric detail, while RGB policies lack 3D geometric awareness; the paper seeks a structured 3D observation usable with standard vision encoders in diffusion imitation learning.
Approach
Represent point clouds as dense 2D XYZ point maps aligned with image pixels (no downsampling), so standard 2D visual encoders can process them directly; fuse per-view point-map and RGB tokens (Cat fusion) and denoise actions with an xLSTM-based EDM diffusion head.
Key results
- On CALVIN ABC→D, PMP reaches an average rollout length of 4.01, above Seer-Large (scratch) at 3.83 among models trained from scratch.
- On RoboCasa (16 tasks, 50 demos), PMP-xyz averages 49.12% success and PMP 47.22%, versus 44.00% for the depth baseline and 40.16% for RGB.
- PMP-xyz runs at 2.9 ms and PMP at 3.9 ms per inference on a single RTX 5080 with ConvNeXt-v2 encoders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use point-map XYZ inputs instead of depth-only inputs (ablation) | PMP-xyz · RoboCasa | average success rate (%): 44.00 -> 49.12 | Helped |
| Fuse RGB and point-map tokens with Cat fusion instead of six-channel early fusion (ablation) | PointMapPolicy · RoboCasa | average success rate (%): 43.12 -> 47.22 | Helped |
| Use XYZ-only point maps instead of RGB-only inputs (ablation) | PointMapPolicy · RoboCasa | average success rate (%): 40.16 -> 49.12 | Helped |
| Use XYZ-only point maps instead of RGB-only inputs (ablation) | PointMapPolicy · CALVIN ABC→D | average rollout length: 3.15 -> 2.03 | Hurt |
| Use fused RGB and point-map inputs rather than XYZ-only point maps (ablation) | PointMapPolicy · RoboCasa | average success rate (%): 49.12 -> 47.22 | Hurt |
| Use fused RGB and point-map inputs rather than XYZ-only point maps (ablation) | PointMapPolicy · CALVIN ABC→D | average rollout length: 2.03 -> 4.01 | Helped |
| Use Cat fusion instead of Add or Attn fusion (ablation) | PointMapPolicy · RoboCasa | success rate | Helped |
| Use ConvNeXt-v2 instead of FiLM-ResNet50 or DaViT for point-map encoding (ablation) | PointMapPolicy · RoboCasa | success rate | Helped |
| Use structured point maps instead of PointNet-xyz, PointNet-color, PointPatch, or 3D-Lifting encoders (ablation) | PMP-xyz with X-Block · RoboCasa | average success rate (%) | Helped |
| Fuse RGB and point-map modalities instead of RGB-only inputs | PointMapPolicy · Real-world Folding | accumulated score: 2.1 -> 2.5 | Helped |
| Use XYZ-only point maps rather than RGB-only inputs on deformable-object tasks (ablation) | PointMapPolicy · Real-world Folding and Sweeping | accumulated score | Hurt |
Surprising findings
- Geometry-only PMP-xyz collapses on CALVIN (2.03 avg length) because the benchmark relies heavily on color instructions, while it is the best variant on RoboCasa.
- Fusing point maps with RGB (PMP) is slightly worse than PMP-xyz on RoboCasa, suggesting most RoboCasa tasks favor geometric information.
Limitations
- Simple concatenation fusion of point map and RGB tokens may not exploit complementary cues optimally.
- Point map encoders are trained from scratch, lacking the ImageNet pretraining benefit RGB encoders receive.
- Point-map-only policy performs poorly on color-dependent tasks such as CALVIN instructions and real-world Cup-Stacking.
Experimental setup
Datasets: CALVIN RoboCasa real-world Franka teleoperation demonstrations
Benchmarks: RoboCasa CALVIN ABC→D real-world six-task Franka evaluation
Metrics: success rate average rollout length task stage score inference latency
Baselines: BC GR00T-N1 DP3 3D Diffuser Actor RGB Depth PMP-6ch RoboFlamingo SuSIE GR-1 OpenVLA RoboDual
Builds on: EDM X-IL DP3 3D Diffuser Actor FiLM-ResNet
Closest papers by meaning
- Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning (NeurIPS 2024 · similarity 0.92)
- PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (NeurIPS 2026 · similarity 0.87)
- What's the Move? Hybrid Imitation Learning via Salient Points (ICLR 2025 · similarity 0.87)
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.87)
- ProxyTransformation: Preshaping Point Cloud Manifold With Proxy Attention For 3D Visual Grounding (CVPR 2025 · similarity 0.86)
- FlowPolicy: Enabling Fast and Robust 3D Flow-Based Policy via Consistency Flow Matching for Robot Manipulation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.