Research map / 3D vision / 3D generation
Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Seg2Reg renders 1D per-column layout depth from a 2D floor-plan density map via differentiable flattened volume rendering, plus a 3D layout warping augmentation and the PanoLayoutStudio benchmarking codebase.
Problem
Single-view 360 room layout reconstruction: regression methods need large 1D decoders and cannot handle occlusion without heuristics, while segmentation methods need non-differentiable post-processing and lose accuracy.
Approach
Reformulate the 2D layout segmentation as a floor-plan density field and apply flattened volume rendering (NeRF-style alpha blending on the 2D density map) to differentiably render 1D layout depth and occluded polygons, trained with a regression-aligned cross-entropy on alpha weights; also a LayoutWarp 3D-based geometric augmentation.
Key results
- On MatterportLayout test with HRNet-18, Seg2Reg reaches 82.83% 3D IoU vs. 82.26% for reproduced LGT-Net, using a 0.015M-parameter decoder.
- On ZInd test, Seg2Reg with HRNet-18 achieves 90.73% 3D IoU vs. 90.61% for reproduced LGT-Net.
- On Stanford2D3D, Seg2Reg achieves 87.24% 3D IoU vs. 85.83% for reproduced LGT-Net, while it is slightly behind on PanoContext (87.23% vs 87.53%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace binary segmentation with a floor-plan density field rendered by Seg2Reg (ablation) | Seg2Reg · MatterportLayout valid | 2D IoU (%): 86.92 -> 87.33 | Helped |
| Replace binary segmentation with a floor-plan density field rendered by Seg2Reg (ablation) | Seg2Reg · MatterportLayout valid | 3D IoU (%): 84.65 -> 85.00 | Helped |
| Replace binary segmentation with a floor-plan density field rendered by Seg2Reg (ablation) | Seg2Reg · ZInd valid | 2D IoU (%): 91.21 -> 92.05 | Helped |
| Replace binary segmentation with a floor-plan density field rendered by Seg2Reg (ablation) | Seg2Reg · ZInd valid | 3D IoU (%): 89.46 -> 90.39 | Helped |
| Add random layout perturbation (ablation) | LGT-Net with ResNet-34 · MatterportLayout valid | 3D IoU (%): 83.32 -> 82.93 | Hurt |
| Add random layout perturbation (ablation) | LGT-Net with ResNet-34 · MatterportLayout valid, irregular | 3D IoU (%): 78.30 -> 78.77 | Helped |
| Add random layout perturbation (ablation) | LGT-Net with ResNet-34, trained on MatterportLayout · ZInd valid (cross-dataset) | 3D IoU (%): 76.14 -> 77.02 | Helped |
| Add random camera-height adjustment (ablation) | LGT-Net with ResNet-34 · MatterportLayout valid | 3D IoU (%): 83.32 -> 83.68 | Helped |
| Add random camera-height adjustment (ablation) | LGT-Net with ResNet-34 · MatterportLayout valid, irregular | 3D IoU (%): 78.30 -> 79.17 | Helped |
| Add random camera-height adjustment (ablation) | LGT-Net with ResNet-34, trained on MatterportLayout · ZInd valid (cross-dataset) | 3D IoU (%): 76.14 -> 76.78 | Helped |
| Use Seg2Reg with HRNet-18 instead of reproduced LGT-Net with HRNet-18 | Room layout reconstruction · MatterportLayout test | 3D IoU (%): 82.26 -> 82.83 | Helped |
| Use Seg2Reg with HRNet-18 instead of reproduced LGT-Net with HRNet-18 | Room layout reconstruction · ZInd test | 3D IoU (%): 90.61 -> 90.73 | Helped |
Surprising findings
- Segmentation-based 2D density methods perform better with HRNet-18 while the regression-based LGT-Net performs better with ResNet-34.
- Random layout perturbation slightly lowers overall 3D IoU on MatterportLayout but improves irregular (non-Manhattan) layouts, revealing a Manhattan bias in models trained on Manhattan-only labels.
Limitations
- Authors have not identified why segmentation-based methods favor certain backbones.
- Layout warping transformations beyond the two instantiated augmentations remain largely unexplored.
- Results on cuboid datasets are slightly behind the reproduced LGT-Net on PanoContext.
Experimental setup
Datasets: MatterportLayout Zillow Indoor (ZInd) PanoContext Stanford2D3D
Benchmarks: MatterportLayout ZInd PanoContext Stanford2D3D
Metrics: 2D IoU 3D IoU
Baselines: HorizonNet HoHoNet LED2-Net LGT-Net LayoutNet v2 DuLaNet v2 AtlantaNet
Builds on: NeRF HorizonNet LGT-Net SegFormer all-MLP decoder PanoStretch RandAug
Closest papers by meaning
- HouseLayout3D: A Benchmark and Training-free Baseline for 3D Layout Estimation in the Wild (NeurIPS 2025 · similarity 0.86)
- No More Ambiguity in 360deg Room Layout via Bi-Layout Estimation (CVPR 2024 · similarity 0.85)
- BADGR: Bundle Adjustment Diffusion Conditioned by Gradients for Wide-Baseline Floor Plan Reconstruction (CVPR 2025 · similarity 0.84)
- MvFFN: Multi-view Floor-Plan Feed-Forward Network for Unposed Wide-Baseline Panorama Layout Reconstruction (NeurIPS 2026 · similarity 0.83)
- IndoorGS: Geometric Cues Guided Gaussian Splatting for Indoor Scene Reconstruction (CVPR 2025 · similarity 0.83)
- CAGE: Continuity-Aware edGE Network Unlocks Robust Floorplan Reconstruction (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.