Research map / 3D vision / 3D generation
Aerial Lifting: Neural Urban Semantic and Building Instance Lifting from Aerial Imagery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lifts noisy 2D semantic and SAM-based building instance labels from aerial images into a NeRF via scale-adaptive label fusion, cross-view instance grouping, and MVS depth priors for urban-scale segmentation.
Problem
Urban-scale semantic and building-level instance segmentation from multi-view aerial images, where objects vary widely in size and 2D labels are multi-view inconsistent.
Approach
Fuses 2D semantic labels across altitudes by rendering far-view images from NeRF and back-projecting building masks, groups SAM instance masks across views using the 3D scene geometry before lifting them to an instance field, and adds MVS depth priors to improve radiance geometry.
Key results
- Scale-adaptive fusion raises 2D building IoU on Yingrenshi from 84.8 (Mask2Former) to 93.2.
- Ours reaches 72.0 mIoU on Yingrenshi versus 32.9 for Panoptic-Lift and 67.6 for Semantic-NeRF (M2F).
- Building PQscene with contrastive learning improves from 54.8 (SAM-Label) to 64.1 on Yingrenshi with cross-view grouping.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use scale-adaptive semantic label fusion instead of the Mask2Former labels alone | Semantic-NeRF (M2F) · Yingrenshi | mIoU: 67.6 -> 72.0 | Helped |
| use scale-adaptive semantic label fusion instead of the Mask2Former labels alone | Semantic-NeRF (M2F) · Yuehai-Campus | mIoU: 70.9 -> 74.9 | Helped |
| use scale-adaptive semantic label fusion instead of the Mask2Former labels alone | Semantic-NeRF (M2F) · Longhua-1 | mIoU: 61.1 -> 66.1 | Helped |
| use scale-adaptive semantic label fusion instead of the Mask2Former labels alone | Semantic-NeRF (M2F) · Longhua-2 | mIoU: 64.1 -> 66.7 | Helped |
| add cross-view grouping to SAM instance labels (ablation) | Contrastive learning instance segmentation · Yingrenshi | PQscene: 54.8 -> 64.1 | Helped |
| add cross-view grouping to SAM instance labels (ablation) | Contrastive learning instance segmentation · Yuehai-Campus | PQscene: 18.8 -> 43.6 | Helped |
| add cross-view grouping to SAM instance labels (ablation) | Contrastive learning instance segmentation · Longhua-1 | PQscene: 29.7 -> 45.8 | Helped |
| add cross-view grouping to SAM instance labels (ablation) | Contrastive learning instance segmentation · Longhua-2 | PQscene: 22.9 -> 31.5 | Helped |
| add geometry-guided instance filtering (ablation) | Linear-assignment instance segmentation with SAM-Label · Yingrenshi | PQscene: 14.4 -> 19.8 | Helped |
| add geometry-guided instance filtering (ablation) | Contrastive-learning instance segmentation with SAM-Label · Yingrenshi | PQscene: 54.8 -> 55.9 | Helped |
| add cross-view grouping after geometry-guided filtering (ablation) | Linear-assignment instance segmentation with SAM-Label · Yingrenshi | PQscene: 19.8 -> 38.7 | Helped |
| add cross-view grouping after geometry-guided filtering (ablation) | Contrastive-learning instance segmentation with SAM-Label · Yingrenshi | PQscene: 55.9 -> 64.1 | Helped |
Surprising findings
- Panoptic-Lift performs poorly on urban aerial scenes (mIoU 20-34) largely due to weak geometry from TensoRF backbone.
- Monocular depth estimators were found not robust for aerial images, especially for nadir views.
Limitations
- Focuses on closed-vocabulary categories (buildings, trees, cars, roads) only.
- Relies on the quality of off-the-shelf 2D segmenters and SAM masks, which over-segment large buildings.
- Per-scene optimization requires lifting labels separately for each scene.
Experimental setup
Datasets: UrbanBIS
Benchmarks: UrbanBIS
Metrics: mIoU PSNR PQscene
Baselines: Mask2Former UNetFormer Panoptic-Lift Semantic-NeRF (M2F) Detectron SAM-Label Contrastive Lifting
Builds on: NeRF Semantic-NeRF Panoptic Lifting Contrastive Lifting Mask2Former
Closest papers by meaning
- Predicting Ground-Level Scene Layout From Aerial Imagery (CVPR 2017 · similarity 0.88)
- UniC-Lift: Unified 3D Instance Segmentation via Contrastive Learning (AAAI 2026 · similarity 0.87)
- GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding (CVPR 2024 · similarity 0.85)
- Learnable Earth Parser: Discovering 3D Prototypes in Aerial Scans (CVPR 2024 · similarity 0.85)
- SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation (CVPR 2024 · similarity 0.84)
- Multi-Level Neural Scene Graphs for Dynamic Urban Environments (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.