Research map / 3D vision / 3D generation
SCube: Instant Large-Scale Scene Reconstruction using VoxSplats
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCube, a feed-forward method that reconstructs large-scale driving scenes from sparse, low-overlap images by generating a sparse-voxel hierarchy with a conditioned latent diffusion model and predicting voxel-supported Gaussians (VoxSplats) plus a sky panorama.
Problem
Reconstructing large-scale, high-resolution 3D scenes (geometry, appearance, semantics) from a few posed, non-overlapping images, where per-scene optimization needs dense views and prior feed-forward methods are low-resolution and blurry.
Approach
Combine a hierarchical sparse-voxel latent diffusion model (XCube), conditioned on DINOv2 image features lifted to 3D via depth-bin unprojection with learned per-pixel depth distributions, with a feed-forward sparse 3D U-Net that predicts M Gaussians per voxel (VoxSplats) and a sky feature panorama for background.
Key results
- On Waymo at the input frame T, SCube reaches 25.90 PSNR vs. 22.15 for PixelSplat and 21.84 for MVSplat; SCube+ reaches 28.01 PSNR.
- At T+5 and T+10 predictions, SCube+ attains 22.32 and 21.09 PSNR, versus best baseline 20.14 and 18.78 (MVSplat).
- As Gaussian splatting initialization on 15 static scenes, SCube initialization gives 24.10 PSNR at R=10 vs. 23.30 for Metric3Dv2 and 21.66 for random.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SCube's per-scene GAN postprocessing | SCube · Waymo novel view synthesis, T / T + 5 / T + 10 | PSNR: 25.90 -> 28.01; 19.90 -> 22.32; 18.78 -> 21.09 | Helped |
| replace depth-weighted image conditioning with broadcasting features along each pixel ray (ablation) | SCube voxel grid reconstruction · Waymo fine-level voxel grid | IoU: 34.31% -> 30.33% | Hurt |
| replace depth-weighted image conditioning with broadcasting features along each pixel ray (ablation) | SCube voxel grid reconstruction · Waymo fine-level voxel grid | mIoU: 20.00% -> 16.61% | Hurt |
| replace two-stage reconstruction with a single-stage model (ablation) | SCube · Waymo novel view synthesis (average of T + 5 and T + 10) | PSNR: 19.34 -> 17.88 | Hurt |
| replace two-stage reconstruction with a single-stage model (ablation) | SCube · Waymo novel view synthesis (average of T + 5 and T + 10) | LPIPS: 0.48 -> 0.57 | Hurt |
| reduce voxel grid resolution from 1024^3 to 256^3 (ablation) | SCube appearance reconstruction · Waymo novel view synthesis | PSNR: 19.34 -> 18.58 | Hurt |
| reduce voxel grid resolution from 1024^3 to 256^3 (ablation) | SCube appearance reconstruction · Waymo novel view synthesis | LPIPS: 0.48 -> 0.62 | Hurt |
| reduce Gaussians per voxel from M = 4 to M = 1 (ablation) | SCube appearance reconstruction · Waymo novel view synthesis | LPIPS: 0.48 -> 0.52 | Hurt |
| add depth supervision to PixelSplat (ablation) | PixelSplat · Waymo prediction (T + 5 and T + 10) | PSNR: 20.11 -> 19.91; 18.77 -> 18.87 | Mixed |
| initialize Gaussian splatting with SCube predictions instead of Metric3Dv2 depth | Gaussian splatting training · Waymo initialization (R = 10, 20, 40) | PSNR: 23.30 -> 24.10; 25.21 -> 25.94; 25.58 -> 26.07 | Helped |
Surprising findings
- Adding depth supervision to PixelSplat hurt performance at T+5 (SSIM 0.70 to 0.58), and adding Metric3Dv2 depth supervision degraded SCube's performance, so depth supervision was dropped.
- Around 80% of ground-truth voxels are occluded from the input views, so geometry is highly uncertain.
Limitations
- Cannot handle complex dynamic scenes under extreme lighting or weather conditions.
- Appearance in occluded regions remains uncertain.
- Requires ground-truth 3D training data, which is not always available for generic outdoor scenes.
Experimental setup
Datasets: Waymo Open Dataset
Benchmarks: Waymo Open Dataset
Metrics: PSNR SSIM LPIPS voxel Chamfer distance IoU mIoU
Baselines: PixelNeRF PixelSplat DUSt3R MVSplat MVSGaussian Metric3Dv2
Builds on: XCube 3D Gaussian Splatting LSS DINOv2 MVDream
Closest papers by meaning
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.88)
- Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.88)
- CompactSplat: Spatially Adaptive Gaussian Distribution for Feedforward Scene Reconstruction (NeurIPS 2026 · similarity 0.88)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.88)
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images (ICLR 2025 · similarity 0.87)
- Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modeling (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.