Research map / Detection and segmentation / Open-set and few-shot detection
IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IPFormer, a vision-based 3D Panoptic Scene Completion method that initializes instance proposals adaptively from visible image-derived 3D context, with a dual-head two-stage training scheme, reporting SOTA in-domain PQ and 14x faster runtime on SemanticKITTI.
Problem
Vision-based 3D Panoptic Scene Completion, where existing Transformer methods use static learned instance queries that do not adapt to the observed scene at test time, and no established camera-based PSC baselines exist.
Approach
Initialize instance queries as context-adaptive instance proposals derived from visible voxels lifted from image features (via visibility-based sampling and 3D deformable attention), so they adapt to each input at train and test time, combined with a dual-head, two-stage (SSC then PSC) training scheme.
Key results
- In-domain on SemanticKITTI val, IPFormer reaches PQ-dagger 14.45 and PQ 6.30, slightly above CGFormer+DBSCAN (14.39 and 6.16).
- Runtime is 0.33 s versus 4.70 s for CGFormer+DBSCAN, a reduction of over 14x.
- Zero-shot to SSCBench-KITTI360, IPFormer reaches PQ 1.23 and PQ-dagger 9.41 versus CGFormer+DBSCAN's 1.08 and 8.44.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace instance queries with context-adaptive instance proposals (ablation) | IPFormer · SemanticKITTI val | PQ-All: 6.08 -> 6.30 (+3.62%) | Helped |
| replace instance queries with context-adaptive instance proposals (ablation) | IPFormer · SemanticKITTI val Thing classes | PQ: 1.76 -> 2.09 (+18.75%) | Helped |
| initialize proposals from visible voxels only rather than visible and invisible voxels (ablation) | IPFormer · SemanticKITTI val Thing classes | SQ: 22.00 -> 42.67 (+93.95%) | Helped |
| remove deep supervision from instance encoding (ablation) | IPFormer · SemanticKITTI val Thing classes | SQ: 21.06 -> 42.67 (+102.61%) | Helped |
| use both dual heads and two-stage training instead of a single-head, single-stage setup (ablation) | IPFormer · SemanticKITTI val | PQ-All: 6.23 -> 6.30 | Mixed |
| replace single-stage training with two-stage training while retaining a single head (ablation) | IPFormer · SemanticKITTI val | PQ-All: 6.23 -> 0.42 | Hurt |
| freeze first-stage weights during stage two (ablation) | IPFormer · SemanticKITTI val Thing classes | RQ: 3.33 -> 0.49 | Hurt |
| add the SSC objective to stage two with λSSC 1.00 (ablation) | IPFormer · SemanticKITTI val | PQ-All: 6.30 -> 6.06 | Hurt |
| add the SSC objective to stage two with λSSC 0.50 (ablation) | IPFormer · SemanticKITTI val | PQ-All: 6.30 -> 6.27 | Hurt |
| add the SSC objective to stage two with λSSC 0.20 (ablation) | IPFormer · SemanticKITTI val | PQ-All: 6.30 -> 6.05 | Hurt |
| add the SSC objective to stage two with λSSC 0.10 (ablation) | IPFormer · SemanticKITTI val | PQ-All: 6.30 -> 6.08 | Hurt |
| add the SSC objective to stage two with λSSC 0.01 (ablation) | IPFormer · SemanticKITTI val | PQ-All: 6.30 -> 5.16 | Hurt |
Surprising findings
- Visible-only proposal sampling outperforms visible+invisible sampling, with SQ-Thing rising by 93.95%.
- Deep supervision of intermediate instance-encoding layers degrades all metrics.
Limitations
- Higher training memory (52.8 GB) than baselines.
- Thing classes with low frequency remain largely unrecognized (e.g., Motorcyclist).
- Cannot be compared directly with PanoSSC due to non-public post-processed data; baselines are DBSCAN-clustered SSC outputs.
Experimental setup
Datasets: SemanticKITTI PaSCo pseudo panoptic labels SSCBench-KITTI360
Benchmarks: SemanticKITTI SSCBench-KITTI360
Metrics: PQ PQ-dagger SQ RQ IoU mIoU runtime
Baselines: MonoScene + DBSCAN Symphonies + DBSCAN OccFormer + DBSCAN CGFormer + DBSCAN
Builds on: CGFormer VoxFormer PaSCo Symphonies DETR
Closest papers by meaning
- PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness (CVPR 2024 · similarity 0.90)
- H2GFormer: Horizontal-to-Global Voxel Transformer for 3D Semantic Scene Completion (AAAI 2024 · similarity 0.90)
- Context and Geometry Aware Voxel Transformer for Semantic Scene Completion (NeurIPS 2024 · similarity 0.89)
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation (AAAI 2025 · similarity 0.89)
- Memory-Augmented Re-Completion for 3D Semantic Scene Completion (AAAI 2025 · similarity 0.88)
- SGFormer: Satellite-Ground Fusion for 3D Semantic Scene Completion (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.