Research map / 3D vision / 3D generation
One at a Time: Progressive Multi-Step Volumetric Probability Learning for Reliable 3D Scene Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VPD, a multi-step volumetric diffusion framework that progressively refines coarse 3D probability volumes from MVS/SSC baselines, with a CACC module and Online Filtering, reaching SOTA on DTU, BlendedMVS, ScanNet and SemanticKITTI.
Problem
Single-step 3D probability volume estimation for multi-view stereo and semantic scene completion struggles in challenging regions such as occlusions, reflections, thin structures and textureless areas.
Approach
Decompose volumetric probability learning into a sequence of conditional diffusion denoising steps, conditioned on a coarse probability volume from an off-the-shelf baseline and multi-scale contextual features; confidence-aware cross-attention (CACC) corrects uncertain regions, and winner-takes-all online filtering keeps the volume unimodal during sampling.
Key results
- On DTU, TransMVSNet + VPD reduces Abs error from 7.21 to 4.94 (about 31.5% reduction).
- On SemanticKITTI test, StereoScene + VPD reaches 16.31 mIoU vs 15.36 for StereoScene, surpassing LiDAR-based SSCNet (16.14).
- Zero-shot DTU-to-BlendedMVS, TransMVSNet + VPD achieves Abs 1.2267 vs 1.9216 for TransMVSNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VPD to TransMVSNet | TransMVSNet · DTU test | Abs: 7.2096 -> 4.9416 | Helped |
| Add VPD to TransMVSNet | TransMVSNet · BlendedMVS validation (zero-shot from DTU) | Abs Rel: 0.0657 -> 0.0376 | Helped |
| Add VPD to ESTD | ESTD · ScanNet test | Abs Rel: 0.0812 -> 0.0753 | Helped |
| Add VPD to StereoScene | StereoScene · SemanticKITTI test | mIoU: 15.36 -> 16.31 | Helped |
| Add CVP (ablation) | CasMVSNet · DTU test | Abs: 7.4381 -> 6.3111 | Helped |
| Add CACC to CVP (ablation) | CasMVSNet + CVP · DTU test | Abs: 6.3111 -> 5.9829 | Helped |
| Add OF to CVP and CACC (ablation) | CasMVSNet + CVP + CACC · DTU test | Abs: 5.9829 -> 5.7275 | Helped |
| Use 1 reverse step instead of 4 (ablation) | TransMVSNet + VPD · DTU test | Abs: 4.94 -> 6.27 | Hurt |
| Use 2 reverse steps instead of 4 (ablation) | TransMVSNet + VPD · DTU test | Abs: 4.94 -> 5.40 | Hurt |
| Use 6 reverse steps instead of 4 (ablation) | TransMVSNet + VPD · DTU test | Abs: 4.94 -> 4.79 | Helped |
| Use 8 reverse steps instead of 4 (ablation) | TransMVSNet + VPD · DTU test | Abs: 4.94 -> 4.72 | Helped |
| Apply CACC and OF directly to the baseline cost volume, without VPD (ablation) | CasMVSNet · DTU test | Abs: 7.4381 -> 7.1218 | Helped |
Surprising findings
- Applying CACC and OF directly to the CasMVSNet cost volume without the diffusion process yields much smaller gains (Abs 7.44 to 7.12) than the full VPD (5.73), which the authors attribute to these modules being effective only within the multi-step generative process.
Limitations
- Runtime and memory increase over baselines (e.g., TransMVSNet 0.87s to 1.24s with 4 steps; memory 8.27 to 10.48 GB).
- Performance gains saturate beyond about 4 reverse steps while cost keeps rising.
Experimental setup
Datasets: DTU BlendedMVS ScanNet SemanticKITTI
Benchmarks: DTU BlendedMVS ScanNet SemanticKITTI
Metrics: Abs Rel Abs Sq Rel Th8 Th20 delta-1 accuracy mIoU
Baselines: MVSNet CasMVSNet UniMVSNet TransMVSNet MonoScene VoxFormer StereoScene SSCNet ESTD
Builds on: CasMVSNet TransMVSNet UniMVSNet StereoScene ESTD DDPM
Closest papers by meaning
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.87)
- BLURD: Benchmarking and Learning using a Unified Rendering and Diffusion Model (NeurIPS 2024 · similarity 0.87)
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.87)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.87)
- HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D (CVPR 2024 · similarity 0.86)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.