Research map / 3D vision / 3D generation
Scene4U: Hierarchical Layered 3D Scene Reconstruction from Single Panoramic Image for Your Immerse Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scene4U builds layered, dynamic-object-free 3D Gaussian Splatting scenes from a single panorama by combining open-vocabulary segmentation, LLM-based layer assignment, diffusion inpainting and depth completion, and per-layer optimization; it also introduces the WorldVista3D landmark panorama dataset.
Problem
Single-view 3D scene reconstruction from panoramas suffers from global texture inconsistency under iterative camera refinement and from voids caused by foreground-background occlusion, while dynamic objects obstruct immersive exploration.
Approach
Decompose a panorama into semantic depth layers (sky, background, foreground, dynamic) using an open-vocabulary segmenter plus an LLM, inpaint each layer back-to-front, lift each layer to a point cloud, and optimize separate 3DGS per layer so occluded background is revealed during free navigation.
Key results
- Scene4U reaches PSNR 32.778, LPIPS 0.025 and BRISQUE 27.793 on WorldVista3D, versus DreamScene360's 29.546 / 0.047 / 34.574.
- Reported improvements over DreamScene360 are 24.24% in LPIPS and 24.40% in BRISQUE, with the fastest training time (11 min 13 s vs 18 min 23 s).
- With 0.3 m camera movement, the layered variant improves NIQE by 5.93% and BRISQUE by 3.35% over the single-layer variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Scene4U instead of DreamScene360 | WorldVista3D scene reconstruction · WorldVista3D | LPIPS: 0.047 -> 0.025 | Helped |
| Use Scene4U instead of DreamScene360 | WorldVista3D scene reconstruction · WorldVista3D | BRISQUE: 34.574 -> 27.793 | Helped |
| Use Scene4U instead of DreamScene360 | WorldVista3D scene reconstruction · WorldVista3D | PSNR: 29.546 -> 32.778 | Helped |
| Use Scene4U instead of DreamScene360 | WorldVista3D scene reconstruction · WorldVista3D | SSIM: 0.922 -> 0.959 | Helped |
| Use Scene4U instead of DreamScene360 | WorldVista3D scene reconstruction · WorldVista3D | training time: 18 min 23 s -> 11 min 13 s | Helped |
| Remove the dynamic-object layer (ablation) | Scene4U · WorldVista3D | PSNR: 31.237 -> 32.778 | Helped |
| Remove the dynamic-object layer (ablation) | Scene4U · WorldVista3D | LPIPS: 0.030 -> 0.025 | Helped |
| Remove the dynamic-object layer (ablation) | Scene4U · WorldVista3D | BRISQUE: 28.892 -> 27.793 | Helped |
| Add Layered Panorama Construction (ablation) | Scene4U · Scene navigation, move 0.1 m | NIQE: 3.015 -> 2.861 | Helped |
| Add Layered Panorama Construction (ablation) | Scene4U · Scene navigation, move 0.1 m | BRISQUE: 31.721 -> 31.539 | Helped |
| Add Layered Panorama Construction (ablation) | Scene4U · Scene navigation, move 0.2 m | NIQE: 3.053 -> 2.930 | Helped |
| Add Layered Panorama Construction (ablation) | Scene4U · Scene navigation, move 0.2 m | BRISQUE: 33.367 -> 32.038 | Helped |
Surprising findings
- LucidDreamer attains higher PSNR (30.409) and SSIM (0.985) than DreamScene360 and Scene4U with dynamic layer removed, though it is worse on LPIPS and BRISQUE.
Limitations
- Evaluation is limited to 120 panoramas from one dataset with no held-out ground truth for novel views, so metrics compare renderings against reference views.
- Layer definitions (dynamic, foreground, background, sky) are predefined and rely on LLM category grouping, which the authors acknowledge is challenging.
Experimental setup
Datasets: WorldVista3D
Benchmarks: WorldVista3D
Metrics: PSNR SSIM LPIPS NIQE BRISQUE training time
Baselines: LucidDreamer DreamScene360 Ours (w/o M-L)
Builds on: 3D Gaussian Splatting LucidDreamer DreamScene360 Instruct Pix2Pix 360MonoDepth
Closest papers by meaning
- 4K4DGen: Panoramic 4D Generation at 4K Resolution (ICLR 2025 · similarity 0.89)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.87)
- Splatter-360: Generalizable 360 Gaussian Splatting for Wide-baseline Panoramic Images (CVPR 2025 · similarity 0.87)
- PanoWorld: Geometry-Consistent Panoramic Video World Modeling (NeurIPS 2026 · similarity 0.87)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.86)
- InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.