Research map / 3D vision / 3D generation
SUGAR: Pre-training 3D Visual Representations for Robotics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SUGAR, a transformer-based 3D point cloud pre-training framework that jointly learns semantics, geometry and affordance from single- and multi-object simulated scenes via five tasks, evaluated on zero-shot 3D recognition, referring expression grounding, and RLBench manipulation.
Problem
Existing visual representation pre-training for robotics is mostly 2D, and existing 3D pre-training is limited to single complete objects, ignoring occlusion, clutter and object affordances.
Approach
A prompt-based transformer (point cloud encoder + prompt decoder) pre-trained with five tasks (masked point modeling, cross-modal distillation, grasping pose synthesis, instance segmentation, referring expression grounding) on automatically generated multi-object simulated scenes with free grasp and segmentation labels, using curriculum learning from single to multi-object data.
Key results
- On Objaverse-LVIS with ensembled pre-training, SUGAR (single) reaches 42.1% Top1 vs 39.1% for OpenShape-PointBERT.
- On OCID-Ref, SUGAR (Ens m) achieves 97.74% total Acc@0.25 vs 90.77% for V-Cond.
- On RLBench 10-task language-guided manipulation, SUGAR (Ens m) reaches 93.0% average success rate vs 83.3% for Hiveformer and 89.8% for PolarNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train on ensembled single-object data instead of ShapeNet-only single-object data | SUGAR · ScanObjectNN OBJ BG | Top1 accuracy: 50.3 -> 68.0 | Helped |
| Further train on multi-object scenes | SUGAR · ScanObjectNN OBJ BG | Top1 accuracy: 68.0 -> 65.6 | Hurt |
| Further train on multi-object scenes | SUGAR · ModelNet40 | Top1 accuracy: 84.3 -> 83.8 | Hurt |
| Use ensembled pre-training data instead of ShapeNet-only data | SUGAR single-object pre-training · Objaverse-LVIS | Top1 accuracy: 27.3 -> 42.1 | Helped |
| Pre-train on ensembled multi-object data | SUGAR referring expression grounding · OCID-Ref test | Acc@0.25: 90.77 -> 97.74 | Helped |
| Increase the key-point count from Ne=768 to Ne=1536 (ablation) | SUGAR without pre-training · RoboRefit testB | Acc@0.5: 50.64 -> 55.62 | Helped |
| Pre-train on single-object data only (ablation) | SUGAR referring expression grounding · RoboRefit testB | Acc@0.5: 55.62 -> 52.90 | Hurt |
| Add multi-object scene pre-training (ablation) | SUGAR referring expression grounding · RoboRefit testB | Acc@0.5: 55.62 -> 61.85 | Helped |
| Add grasping pose synthesis to pre-training (ablation) | SUGAR referring expression grounding · RoboRefit testB | Acc@0.5: 59.70 -> 61.85 | Helped |
| Add ensembled multi-object pre-training data (ablation) | SUGAR referring expression grounding · RoboRefit testB | Acc@0.5: 61.85 -> 65.04 | Helped |
| Pre-train on ShapeNet single-object data (ablation) | SUGAR manipulation policy · RLBench 10 tasks | Avg. success rate: 85.9 -> 88.1 | Helped |
| Pre-train on ShapeNet multi-object scenes instead of single-object data (ablation) | SUGAR manipulation policy · RLBench 10 tasks | Avg. success rate: 88.1 -> 93.0 | Helped |
Surprising findings
- SUGAR (multi) underperforms SUGAR (single) by about 3% on several zero-shot recognition benchmarks when pre-training data grows, which the authors attribute to limited model capacity and Objaverse multi-level scenes.
- Pre-training on single-object data alone (SN s) does not help referring expression grounding on the harder RoboRefit testB split.
Limitations
- Small model capacity may limit joint learning of five pre-training tasks at larger data scales.
- Objaverse scenes include multi-level buildings that are not well handled by the simple scene construction.
- Grasp labels only cover ShapeNet objects via ACRONYM, limiting affordance supervision to that domain.
Experimental setup
Datasets: ShapeNet 3D-FUTURE ABO Objaverse ACRONYM ModelNet40 ScanObjectNN Objaverse-LVIS OCID-Ref RoboRefit RLBench
Benchmarks: ModelNet40 ScanObjectNN Objaverse-LVIS OCID-Ref RoboRefit RLBench
Metrics: Top1 accuracy Top3 accuracy Top5 accuracy Acc@0.25 Acc@0.5 mIoU success rate
Baselines: ReCon CLIP2Point ULIP-PointBERT OpenShape-PointBERT R3M MVP CLIP V-Cond RefTR Hiveformer PolarNet Auto-λ
Builds on: OpenShape ReCon Hiveformer DETR PointNet
Closest papers by meaning
- 3D-MVP: 3D Multiview Pretraining for Manipulation (CVPR 2025 · similarity 0.87)
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.87)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.86)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.86)
- Pre-training Auto-regressive Robotic Models with 4D Representations (ICML 2025 · similarity 0.86)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.