Research map / Detection and segmentation / 3D detection and driving
GrabS: Generative Embodied Agent for 3D Object Segmentation without Scene Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GrabS, a two-stage unsupervised 3D object segmentation pipeline that learns generative (VAE/diffusion) SDF object priors from ShapeNet and trains an RL embodied "dynamic container" agent, rewarded by those priors, to discover objects in scenes, then distills pseudo-masks into a Mask3D-style segmenter.
Problem
Segmenting complex 3D objects in point clouds without human 3D scene labels; existing unsupervised methods mostly find simple objects (e.g., cars) or produce poor masks because pretrained 2D features lack objectness.
Approach
Learn generative object-centric priors (orientation estimator + VAE/diffusion SDF decoder) from single ShapeNet objects, then use them as a reward model for an RL agent (cylindrical container with PPO) that explores scenes and proposes object masks used as pseudo-labels to train a segmentation network; the agent is discarded at inference.
Key results
- On ScanNet validation (chair), GrabS (Ours-VAE) reaches 46.7 AP / 71.5 AP50, versus 24.6 AP for EFEM and 37.2 AP for Unscene3D among unsupervised methods.
- On the ScanNet hidden test set, GrabS (Ours-VAE) achieves 29.0 AP vs 20.2 for EFEM among unsupervised methods.
- On the synthetic 6-class dataset, GrabS (Ours-VAE) achieves 58.7 AP vs 20.7 for EFEM and 7.6 for HDBSCAN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the VAE latent distribution with a deterministic AE latent vector (ablation) | GrabS · ScanNet val | AP(%): 46.7 -> 32.0 | Hurt |
| Add the object orientation estimator (ablation) | GrabS · ScanNet val | AP(%): 35.3 -> 46.7 | Helped |
| Add the object discovery branch (ablation) | GrabS · ScanNet val | AP(%): 34.2 -> 46.7 | Helped |
| Add the object segmentation branch (ablation) | GrabS · ScanNet val | AP(%): 25.7 -> 46.7 | Helped |
| Use Diffusion instead of VAE object priors | GrabS · ScanNet val | AP(%): 46.7 -> 47.1 | Mixed |
| Use Diffusion instead of VAE object priors | GrabS · S3DIS Area-5 | AP(%): 46.4 -> 44.2 | Hurt |
| Use Diffusion instead of VAE object priors | GrabS · Synthetic test | AP(%): 58.7 -> 58.5 | No clear effect |
| Remove attention blocks (ablation) | GrabS object-centric network · ScanNet val | AP(%): 46.7 -> 42.9 | Hurt |
| Use 25 rather than 50 parallel trajectories (ablation) | GrabS object discovery branch · ScanNet val | AP(%): 46.7 -> 42.0 | Hurt |
| Use 75 rather than 50 parallel trajectories (ablation) | GrabS object discovery branch · ScanNet val | AP(%): 46.7 -> 46.9 | No clear effect |
| Use SPG superpoints instead of ScanNet superpoints (ablation) | GrabS segmentation branch · ScanNet val | AP(%): 46.7 -> 43.7 | Hurt |
| Remove superpoints and use voxels (ablation) | GrabS segmentation branch · ScanNet val | AP(%): 46.7 -> 45.1 | No clear effect |
Surprising findings
- Class-agnostic evaluation on ScanNet and S3DIS shows GrabS underperforms Part2Object (2D only) and Unscene3D (2D+3D), which use large 2D DINO features, though it beats Unscene3D (3D only).
- Replacing the VAE with a deterministic AE drops AP from 46.7 to 32.0 on ScanNet validation, indicating generative priors matter most.
Limitations
- Performance gap between ScanNet validation and hidden test sets, attributed to distribution shift.
- Object-centric priors come from only six ShapeNet classes, limiting open-vocabulary generality versus DINO-based methods.
- Failure cases include segmenting wall parts as chairs and missing heavily occluded chairs.
Experimental setup
Datasets: ShapeNet ScanNet S3DIS
Benchmarks: ScanNet validation ScanNet hidden test S3DIS Area-5 cross-dataset S3DIS all 6 areas GrabS synthetic test set
Metrics: AP AP50 AP25 recall precision
Baselines: EFEM Unscene3D Part2Object HDBSCAN Mask3D OpenIns3D SAI3D EFEM_mask3d
Builds on: EFEM Mask3D Diffusion-SDF PPO
Closest papers by meaning
- ODIN: A Single Model for 2D and 3D Segmentation (CVPR 2024 · similarity 0.86)
- EmbodiedSAM: Online Segment Any 3D Thing in Real Time (ICLR 2025 · similarity 0.86)
- Online Segment Any 3D Thing as Instance Tracking (NeurIPS 2025 · similarity 0.86)
- Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes (ICML 2025 · similarity 0.86)
- AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation (ICLR 2024 · similarity 0.85)
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.