Research map / Point clouds / Point cloud classification
Online Segment Any 3D Thing as Instance Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Recasts online 3D instance segmentation as instance tracking, using long-term and short-term object-query memory plus spatial consistency learning to counter VFM (SAM) over-segmentation.
Problem
Online, real-time, fine-grained 3D instance segmentation from streaming RGB-D frames, where VFM-derived masks are fragmented/over-segmented and prior query-based pipelines lack instance-level temporal modeling.
Approach
Treats each VFM-derived mask as a track query and propagates instance-level embeddings over time via confidence-gated Hungarian association with a bounded long-term track bank, distance-aware short-term cross-frame attention, and one-to-many fragment supervision with learned mask merging at inference.
Key results
- On ScanNet200 class-agnostic with SAM, AutoSeg3D reaches 45.5 AP / 66.7 AP50 vs. ESAM 42.2 / 63.7 at 0.7 FPS.
- With FastSAM, AutoSeg3D achieves 46.2 AP on ScanNet200 at 10.1 FPS, versus ESAM-E 43.4 AP at 10.6 FPS.
- Trained on ScanNet200 and transferred to 3RScan, AutoSeg3D reaches 16.0 AP vs. ESAM 14.1 AP; on ScanNet it gains 1.8 AP over ESAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoSeg3D instead of ESAM | Online 3D instance segmentation with SAM · ScanNet200 | AP: 42.2 -> 45.5 | Helped |
| Use AutoSeg3D instead of ESAM-E | Online 3D instance segmentation with FastSAM · ScanNet200 | AP: 43.4 -> 46.2 | Helped |
| Use AutoSeg3D instead of ESAM | Online 3D instance segmentation · ScanNet | AP: 41.6 -> 43.4 | Helped |
| Use AutoSeg3D instead of ESAM | Online 3D instance segmentation trained on ScanNet · SceneNN | AP: 30.3 -> 33.1 | Helped |
| Add LTM to the no-memory baseline (ablation) | AutoSeg3D · ScanNet200 component ablation | AP: 41.6 -> 44.1 | Helped |
| Add STM to the no-memory baseline (ablation) | AutoSeg3D · ScanNet200 component ablation | AP: 41.6 -> 42.9 | Helped |
| Add ICMS to LTM + STM (ablation) | AutoSeg3D · ScanNet200 component ablation | AP: 44.8 -> 45.6 | Helped |
| Add LMI to LTM + STM (ablation) | AutoSeg3D · ScanNet200 component ablation | AP: 44.8 -> 45.5 | Helped |
| Add geometric cues to LTM (ablation) | Long-Term Memory · ScanNet200 LTM ablation | AP: 43.2 -> 43.7 | Helped |
| Add confidence estimation and recall to LTM (ablation) | Long-Term Memory · ScanNet200 LTM ablation | AP: 43.2 -> 46.2 (+3.0) | Helped |
| Add cross-frame attention to STM (ablation) | Short-Term Memory · ScanNet200 STM ablation | AP: 44.7 -> 45.1 | Helped |
| Add distance-aware attention and query-specific scales to STM (ablation) | Short-Term Memory · ScanNet200 STM ablation | AP: 44.7 -> 46.2 (+1.5) | Helped |
Surprising findings
- Applying learning-based mask integration during training degrades performance (44.5 vs 45.6 AP), attributed to early-stage false fusions.
- Naively replacing one-to-one loss with one-to-many supervision lowers accuracy; a single-branch configuration drops AP to 44.2 versus 46.2 for dual-branch.
Limitations
- Does not explicitly model relative motion of moving objects.
- Mask integration during training hurts convergence due to early-stage inaccurate fusions.
Experimental setup
Datasets: ScanNet ScanNet200 ScanNet(200)-25k
Benchmarks: ScanNet200 ScanNet SceneNN 3RScan
Metrics: AP AP50 AP25 FPS
Baselines: ESAM SAM3D SAMPro3D Open3DIS SAI3D INS-Conv TD3D-MA TD3D Oneformer3D
Builds on: ESAM SAM Mask3D Hungarian algorithm
Closest papers by meaning
- EmbodiedSAM: Online Segment Any 3D Thing in Real Time (ICLR 2025 · similarity 0.91)
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.87)
- AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation (ICLR 2024 · similarity 0.87)
- ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation (CVPR 2026 · similarity 0.86)
- Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.86)
- ODIN: A Single Model for 2D and 3D Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.