Research map / Point clouds / Point cloud classification
MirageRoom: 3D Scene Segmentation with 2D Pre-trained Models by Mirage Projection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MirageRoom, which uses a parametric mirage projection (heterogeneous refractive medium) to turn 3D indoor point clouds into multi-view 2D images that feed a frozen Swin-Tiny backbone, fused into 3D point features via a masked reprojection cross-attention for point cloud segmentation.
Problem
3D indoor scene point cloud segmentation suffers from occlusion and incoherence when points are projected to 2D with straight-line projection, limiting the use of 2D pre-trained models without extra real images or camera poses.
Approach
Replace straight-line projection (homogeneous medium) with mirage projection that bends rays along parabolic trajectories via a single coefficient κ (height offset κy²), exposing occluded points; a masked 2D-to-3D cross-attention then fuses 2D features from frozen pre-trained backbones into point features.
Key results
- MirageRoom reaches 72.0% mIoU on S3DIS Area 5, above PTv2 (71.6%) and PointNeXt (70.5%).
- On ScanNet V2 validation, MirageRoom achieves 74.9% mIoU using 3D input only, above BPNet (73.9%) which uses images and poses, and below PTv2 (75.4%) with about 60% fewer parameters.
- Mirage projection raises 3D point occupancy from 20.0% to 35.2% on S3DIS and from 47.9% to 73.9% on ScanNet V2 versus straight-line projection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mirage projection (kappa 0,1,2) instead of straight-line projection for 2D model input (ablation) | MirageRoom with 2D pre-trained backbone · S3DIS Area 5 | mIoU (%): 69.5 -> 72.0 | Helped |
| Add 2D pre-trained model guidance (vs no 2D model) (ablation) | MirageRoom · S3DIS Area 5 | mIoU (%): 70.4 -> 72.0 | Helped |
| Naive straight-line projection with 2D model hurts relative to no 2D model (ablation) | MirageRoom · S3DIS Area 5 | mIoU (%): 70.4 -> 69.5 | Hurt |
| Increase group of kappa from single kappa 0 to {0,1,2} (ablation) | mirage projection group · S3DIS Area 5 | mIoU (%): 70.9 -> 72.0 | Helped |
| Use large kappa value 3 in the group (0,1,3) instead of 2 (ablation) | mirage projection group · S3DIS Area 5 | mIoU (%): 72.0 -> 71.6 | Hurt |
| Add mask in masked reprojection module (ablation) | masked reprojection (k=1) · S3DIS Area 5 | mIoU (%): 70.7 -> 71.8 | Helped |
| Increase neighbour count k from 1 to 5 in masked cross-attention (ablation) | masked reprojection module · S3DIS Area 5 | mIoU (%): 71.8 -> 72.0 | Helped |
| Increase neighbour count k from 5 to 9 (ablation) | masked reprojection module · S3DIS Area 5 | mIoU (%): 72.0 -> 71.9 | Hurt |
| Mirage projection covers more 3D points than straight-line projection | projection occupancy · S3DIS | occupancy (%): 20.0 -> 35.2 | Helped |
| Mirage projection covers more 3D points than straight-line projection | projection occupancy · ScanNet V2 | occupancy (%): 47.9 -> 73.9 | Helped |
| Use MirageRoom (3D-only input with 2D pretrained guidance) vs PTv2 with more parameters on ScanNet V2 | MirageRoom · ScanNet V2 val | mIoU (%): 74.9 vs 75.4 (PTv2), with about 60% fewer parameters | Mixed |
Surprising findings
- Straight-line projection with a 2D model (69.5% mIoU) performs worse than the no-2D-model baseline (70.4%) on S3DIS.
Limitations
- Complete solution to occlusion remains open, as stated by the authors.
- The potential of mirage projection is not fully explored.
Experimental setup
Datasets: S3DIS ScanNet V2 ImageNet-1K
Benchmarks: S3DIS Area 5 ScanNet V2 validation
Metrics: mIoU mAcc OA occupancy
Baselines: PTv1 PTv2 PointNeXt PMeta StraFormer BPNet MVPNet KPConv MinkUNet
Builds on: Swin Transformer Point Transformer V2 Feature Pyramid Network
Closest papers by meaning
- Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition (AAAI 2024 · similarity 0.86)
- LiDAR-Net: A Real-scanned 3D Point Cloud Dataset for Indoor Scenes (CVPR 2024 · similarity 0.86)
- ComPC: Completing a 3D Point Cloud with 2D Diffusion Priors (ICLR 2025 · similarity 0.85)
- UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting (CVPR 2025 · similarity 0.84)
- Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding (CVPR 2024 · similarity 0.84)
- On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.