Research map / Detection and segmentation / 3D detection and driving
SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGFormer, a LiDAR-camera fusion transformer for 3D panoptic segmentation that adds context-guided image feature aggregation and a semantic-geometry transformer, reaching SOTA on nuScenes and SemanticPOSS.
Problem
Multi-modal (LiDAR + camera) 3D panoptic segmentation for autonomous driving, where image semantic features treat all pixels equally and fused representations lack comprehensive semantic and geometric information.
Approach
Adaptive semantic-context aggregation (context-guided pixel-context attention, multi-level spatial context alignment, point-pixel refinement) feeds a semantic-geometry transformer that fuses point-pixel geometry features with learned semantic category embeddings via self- and cross-attention.
Key results
- Achieves 80.9 PQ on nuScenes validation, surpassing LCPS (79.8 PQ) and reaching 81.3 mIoU.
- Reaches 64.2 PQ on SemanticKITTI validation, a 5.2% PQ gain over LCPS (59.0).
- Improves over LCPS by 3.4% PQ on the FoV-limited robustness setting and 2.4% on camera occlusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Context-Guided Attention (CGA) (ablation) | SGFormer image branch · nuScenes validation | PQ: 80.0 -> 80.9 | Helped |
| add Spatial Context Alignment (SCA) (ablation) | SGFormer image branch · nuScenes validation | PQ: 79.8 -> 80.9 | Helped |
| add Point-Pixel Refinement (PPR) (ablation) | SGFormer image branch · nuScenes validation | PQ: 80.6 -> 80.9 | Helped |
| replace CGA with Spatial-Channel Enhancement (ablation) | SGFormer image branch · nuScenes validation | PQ: 80.3 -> 80.9 | Hurt |
| replace SCA gated-fusion input with 3DCVF (ablation) | SGFormer image branch · nuScenes validation | PQ: 80.3 -> 80.9 | Hurt |
| replace SCA gated-fusion input with LiraFusion (ablation) | SGFormer image branch · nuScenes validation | PQ: 80.6 -> 80.9 | Hurt |
| add geometry aggregation (ablation) | SGFormer transformer · nuScenes validation | PQ: 80.0 -> 80.9 | Helped |
| add semantic learning (ablation) | SGFormer transformer · nuScenes validation | PQ: 79.6 -> 80.9 | Helped |
| replace semantic-geometry fusion with feature concatenation (ablation) | SGFormer transformer · nuScenes validation | PQ: 77.1 -> 80.9 | Hurt |
| replace semantic-geometry fusion with cross-attention (ablation) | SGFormer transformer · nuScenes validation | PQ: 78.2 -> 80.9 | Hurt |
| increase semantic-geometry fusion layers from two to three (ablation) | SGFormer · nuScenes validation | PQ: 80.9 -> 80.6 | Hurt |
| add ASCA to the baseline image branch (ablation) | SGFormer architecture · nuScenes validation | PQ: +2.5% | Helped |
Surprising findings
- Increasing the number of semantic-geometry fusion layers from 2 to 3 slightly decreases PQ (80.9 to 80.6).
- On SemanticKITTI, the method is only competitive with PUPS on PQ, while PUPS is reported as state-of-the-art there.
Limitations
- Performance on SemanticKITTI is weaker because only two front-view cameras limit point-image matching.
- Adding a third semantic-geometry fusion layer reduces accuracy, indicating redundancy or information loss in deeper stacks.
Experimental setup
Datasets: nuScenes SemanticKITTI SemanticPOSS Cityscapes PASCAL VOC CamVid nuScenes-C SemanticKITTI-C
Benchmarks: nuScenes SemanticKITTI SemanticPOSS nuScenes-R robustness Cityscapes PASCAL VOC CamVid SemanticKITTI-C nuScenes-C
Metrics: PQ PQ† SQ RQ mIoU mCE mRR
Baselines: LCPS CFNet Panoptic PH-Net PUPS 4D-Former Panoptic-PolarNet GP-S3Net EfficientLPS CPSeg SCAN Cylinder3D 2DPASS
Builds on: Cylinder Convolution LCPS Panoptic-PolarNet ResNet-18 FPN
Closest papers by meaning
- IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals (NeurIPS 2025 · similarity 0.89)
- H2GFormer: Horizontal-to-Global Voxel Transformer for 3D Semantic Scene Completion (AAAI 2024 · similarity 0.87)
- Multi-view Consistent 3D Panoptic Scene Understanding (AAAI 2025 · similarity 0.86)
- 3D Vision-Language Gaussian Splatting (ICLR 2025 · similarity 0.86)
- AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation (NeurIPS 2026 · similarity 0.86)
- PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.