Research map / Detection and segmentation / 3D detection and driving
How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IAL, a transformer-based LiDAR-camera 3D panoptic segmentation framework with modality-synchronized augmentation (PieAug), geometry-guided token fusion (GTF), and prior-based query generation (PQG), achieving SOTA PQ on nuScenes and SemanticKITTI.
Problem
LiDAR-based 3D panoptic segmentation struggles with sparse returns for distant or small objects; existing LiDAR-camera methods suffer from LiDAR-image augmentation misalignment and rely on post-processing for panoptic inference.
Approach
Synchronize LiDAR and image augmentation via pie-shaped cylindrical voxel regions paired with image patches; fuse voxel and image features as tokens using geometry-guided projection and scale-aware positional embedding; initialize instance queries from geometric priors (LiDAR heatmap), texture priors (Grounding-DINO + SAM masks lifted to 3D), and learnable no-prior queries, feeding a transformer d
Key results
- On nuScenes validation, IAL reaches 82.3 PQ versus 79.8 for LCPS (L+C), a 2.5-point gain.
- On nuScenes test, IAL reaches 82.0 PQ, ahead of LCPS at 79.5 and 4DFormer at 78.0.
- On SemanticKITTI validation, IAL reaches 63.1 PQ versus 59.0 for LCPS, a 4.1-point gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PieAug to basic point cloud transformations (ablation) | IAL · nuScenes val | PQ: 75.7 -> 78.4 (+2.7%) | Helped |
| add GTF to PieAug (ablation) | IAL · nuScenes val | PQ: 78.4 -> 81.1 (+2.7%) | Helped |
| add PQG to PieAug and GTF (ablation) | IAL · nuScenes val | PQ: 81.1 -> 82.3 (+1.2%) | Helped |
| combine geometric-prior, texture-prior, and no-prior queries (ablation) | IAL PQG · nuScenes val | PQ: 80.7 -> 82.3 | Helped |
| remove no-prior queries (ablation) | IAL PQG with geometric- and texture-prior queries · nuScenes val | PQ: 82.3 -> 80.7 | Hurt |
| add geometric-prior queries to no-prior queries (ablation) | IAL PQG · nuScenes val | PQ: 81.2 -> 81.3 | No clear effect |
| add texture-prior queries to no-prior queries (ablation) | IAL PQG · nuScenes val | PQ: 81.2 -> 81.1 | No clear effect |
| synchronize image augmentation with LiDAR in PieAug (ablation) | PieAug · nuScenes val | PQ: 81.4 -> 82.3 | Helped |
| replace virtual-center token selection with physical-point selection (ablation) | GTF with no positional embedding · nuScenes val | PQ: 78.4 -> 79.3 | Helped |
| add scale-aware positional embedding with physical-point token selection (ablation) | GTF · nuScenes val | PQ: 79.3 -> 81.1 | Helped |
| replace scale-aware positional embedding with center positional embedding (ablation) | GTF with physical-point token selection · nuScenes val | PQ: 81.1 -> 80.6 | Hurt |
| replace scale-aware positional embedding with extreme-point positional embedding (ablation) | GTF with physical-point token selection · nuScenes val | PQ: 81.1 -> 80.0 | Hurt |
Surprising findings
- The LiDAR-only branch of IAL reaches 77.0 PQ on nuScenes validation, already above LCPS's LiDAR-only 72.9 PQ.
- Adding ground-truth center positions to thing queries raises PQ by 6.2 points, motivating better query initialization.
Limitations
- Texture-prior queries rely on generic pre-trained Grounding-DINO and SAM rather than task-specific models.
- The query sampling in PQG uses a relatively simple generic approach (FPS).
- Full model with 2D mask pre-processing is slow (0.9 FPS) and large (859.9M params).
Experimental setup
Datasets: nuScenes SemanticKITTI
Benchmarks: nuScenes SemanticKITTI
Metrics: PQ PQ† RQ SQ PQ^th PQ^st mIoU
Baselines: LCPS Panoptic-FusionNet P3Former CenterLPS P.-PHNet 4DFormer LidarMultiNet MaskPLS
Builds on: P3Former Cylinder3D LCPS PolarMix LaserMix
Closest papers by meaning
- Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation (AAAI 2024 · similarity 0.87)
- Zero-Shot 4D Lidar Panoptic Segmentation (CVPR 2025 · similarity 0.87)
- Multi-view Consistent 3D Panoptic Scene Understanding (AAAI 2025 · similarity 0.87)
- AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis (CVPR 2024 · similarity 0.86)
- Multi-Space Alignments Towards Universal LiDAR Segmentation (CVPR 2024 · similarity 0.86)
- PC-BEV: An Efficient Polar-Cartesian BEV Fusion Framework for LiDAR Semantic Segmentation (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.