Research map / 3D vision / Human pose estimation
Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OBSBench, a benchmark over ManiSkill2 and RLBench to compare RGB, RGB-D, and point-cloud observation spaces for robot imitation learning, finding point clouds generally outperform image-based inputs from scratch and with pre-training.
Problem
Robot learning policies are usually built on RGB images, but the choice of observation modality (RGB, RGB-D, point cloud) is rarely compared under a fair, unified setup, so its effect on manipulation performance, generalization, and sample efficiency is unclear.
Approach
A unified benchmark with standardized encoders (ResNet, ViT, MultiViT, PointNet, SpUNet) and policies (ACT, Diffusion Policy) that varies only the observation modality, plus systematic tests of pre-trained representations, camera-view and visual shifts, sample efficiency, and point-cloud design choices (sampling order, coordinate/color features, pointmaps).
Key results
- Point cloud encoders attain the best mean success rate and mean rank; with diffusion policy, SpUNet and PointNet beat the best other modality by 53.85% and 76.92% in mean S.R.
- Under zero-shot 5-10 degree camera view shifts, PointNet averages 0.40 mean S.R. versus 0.18 for ResNet RGB and 0.09 for ViT-B.
- Real-world experiments on a low-cost bimanual robot: point cloud reaches 0.80 on Reach Cube, 0.40 on Pick Cube and 0.80 on Fold Cloth, versus RGB 0.60/0.05/0.65.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RGB or RGB-D observations with point clouds | ACT and Diffusion Policy · OBSBench 19 tasks | Mean S.R. and Mean Rank: 0.32 -> 0.39 Mean S.R.; 3.05 -> 2.15 Mean Rank | Helped |
| add depth to RGB input | RGB-D ResNet · OBSBench 19 tasks, ACT | Mean Rank: 3.05 -> 3.15 | Hurt |
| use PonderV2 pre-training | SpUNet · OBSBench RLBench | Mean S.R.: 0.37 -> 0.43 | Helped |
| replace image observations with point clouds | robot policies · Zero-shot camera view changes | Average Mean S.R.: 0.18 -> 0.40 | Helped |
| use point cloud observations instead of other modalities | robot policies · RLBench sample efficiency | Mean S.R. | No clear effect |
| use pre-trained visual representations | robot policies · RLBench sample efficiency | Mean S.R.: 0.324 -> 0.439 | Helped |
| use post-sampling instead of pre-sampling (ablation) | point cloud methods · ManiSkill2 tasks, ACT | Mean S.R.: 0.21 -> 0.41 SpUNet; 0.31 -> 0.45 PointNet | Helped |
| add coordinate features (ablation) | post-sampled PointNet · ManiSkill2 tasks, ACT | Mean S.R.: 0.22 -> 0.45 | Helped |
| add color features (ablation) | post-sampled PointNet · ManiSkill2 tasks, ACT | Mean S.R.: 0.38 -> 0.45 | Helped |
| use pointmap observations instead of RGB images (ablation) | ResNet and ViT policies · ManiSkill2 tasks, ACT | Mean S.R.: 0.41 -> 0.43 ResNet; 0.19 -> 0.28 ViT | Helped |
| replace diffusion UNet with Transformer (ablation) | PointNet diffusion policy · PickCube | Mean S.R.: 0.900 -> 0.915 | Helped |
| replace diffusion UNet with Transformer (ablation) | SpUNet diffusion policy · PickCube | Mean S.R.: 0.740 -> 0.527 | Hurt |
Surprising findings
- Depth information generally degrades performance across all settings, including depth-only and channel-stacked RGB-D inputs.
- PonderV2, pre-trained on only about 4.5K point clouds, gives a larger gain than R3M (5M images) and is comparable to VC-1 and MultiMAE.
Limitations
- Point cloud methods show weaker sample efficiency than other modalities in low-data regimes.
- Experiments are mainly in simulation on 19 selected tasks from a 125-task benchmark, with real-world validation on only three tasks.
- Pointmaps underperform point clouds, possibly because encoders for pointmaps are underexplored.
Experimental setup
Datasets: ManiSkill2 demonstrations RLBench demonstrations PonderV2 pre-training data R3M pre-training data VC-1 pre-training data MultiMAE pre-training data
Benchmarks: OBSBench (ManiSkill2 tasks) OBSBench (RLBench tasks)
Metrics: success rate mean success rate mean rank
Baselines: ResNet ViT-B MultiViT PointNet SpUNet R3M VC-1 MultiMAE PonderV2
Builds on: Action Chunking Transformer Diffusion Policy PonderV2 MultiMAE VC-1 R3M
Closest papers by meaning
- PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning (NeurIPS 2025 · similarity 0.92)
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.87)
- PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (NeurIPS 2026 · similarity 0.87)
- Sonata: Self-Supervised Learning of Reliable Point Representations (CVPR 2025 · similarity 0.87)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.86)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.