Research map / Point clouds / Segmentation and detection
AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AGILE3D, an attention-based interactive model that segments multiple 3D objects jointly from point-cloud clicks, using click-as-query encoding, click attention, and iterative multi-object training.
Problem
Interactive 3D point-cloud segmentation has been done one object at a time with binary click maps, ignoring synergies between adjacent objects and costing a full network pass per object per click round.
Approach
Encode user clicks as spatial-temporal queries that interact with each other and with the scene via click-to-scene, click-to-click and scene-to-click attention, fuse click queries into a holistic multi-region mask, and train with iterative multi-object click simulation so only a lightweight decoder runs per click.
Key results
- Single-object ScanNet40 training: IoU@5 79.9 vs 72.4 for InterObject3D and 78.0 for InterObject3D++ (per Table 1); KITTI-360 IoU@5 44.4 vs 19.9 for InterObject3D++.
- Multi-object ScanNet40: reaches 80% average IoU with about 4 fewer clicks than InterObject3D and 2 fewer than InterObject3D++.
- Inference about 2x faster than baselines (0.5s vs 1.2s at 5 clicks per object) with backbone features precomputed once per scene.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AGILE3D instead of InterObject3D++ | ScanNet40 single-object segmentation (ScanNet40 → ScanNet40) · ScanNet40 | IoU@5: 78.0 -> 79.9 | Helped |
| use AGILE3D instead of InterObject3D++ | single-object segmentation · S3DIS-A5 | IoU@5: 80.8 -> 83.5 | Helped |
| use AGILE3D instead of InterObject3D++ | single-object segmentation · KITTI-360 | IoU@5: 19.9 -> 44.4 | Helped |
| use AGILE3D instead of InterObject3D++ | multi-object segmentation · ScanNet40 | mean IoU@mean 10: 82.6 -> 85.0 | Helped |
| use AGILE3D instead of InterObject3D++ | multi-object segmentation · S3DIS-A5 | mean IoU@mean 10: 88.3 -> 88.3 | No clear effect |
| replace random click sampling with iterative training (ablation) | AGILE3D multi-object segmentation · ScanNet40 | mean IoU@mean 10: 82.0 -> 84.4 | Helped |
| add C2S attention (ablation) | AGILE3D · ScanNet40 multi-object | mean IoU@mean 10: 82.8 -> 84.4 | Helped |
| add C2C attention (ablation) | AGILE3D · KITTI-360 multi-object | mean IoU@mean 5: 33.8 -> 37.4 | Helped |
| add S2C attention (ablation) | AGILE3D · ScanNet40 multi-object | mean IoU@mean 10: 83.1 -> 84.4 | Helped |
| add spatial click encoding (ablation) | AGILE3D · ScanNet40 multi-object | mean IoU@mean 10: 84.2 -> 84.4 | Helped |
| add temporal click encoding (ablation) | AGILE3D · ScanNet40 multi-object | mean IoU@mean 10: 83.9 -> 84.4 | Helped |
| use late fusion with max instead of late fusion with mean (ablation) | AGILE3D query fusion · ScanNet40 multi-object | mean IoU@mean 10: 83.8 -> 84.4 | Helped |
Surprising findings
- Simply adding click sharing to the baselines gives only marginal gains or even hurts InterObject3D.
Limitations
- Struggles with object-part segmentation (e.g., chair seat vs whole chair) and ambiguity of user intent.
- Does not predict semantic labels along with masks.
- Failure cases on fine-grained part segmentation.
Experimental setup
Datasets: ScanNetV2 S3DIS KITTI-360 ARKitScenes
Benchmarks: ScanNetV2-Val (ScanNet20/ScanNet40) S3DIS Area 5 KITTI-360
Metrics: IoU@k NoC@q mean IoU over objects (multi-object) inference time
Baselines: InterObject3D InterObject3D++ Mask3D ITIS RITM
Builds on: InterObject3D RITM ITIS Mask3D Minkowski Engine
Closest papers by meaning
- Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes (ICML 2025 · similarity 0.89)
- COS3D: Collaborative Open-Vocabulary 3D Segmentation (NeurIPS 2025 · similarity 0.88)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.88)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.88)
- X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition (CVPR 2024 · similarity 0.87)
- Order-aware Interactive Segmentation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.