Research map / 3D vision / 3D generation
View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes reconstruction error-guided view selection, with Zero-1-to-3 view synthesis and iterative fine-tuning, to improve view transformation robustness of voxel-based multi-view 3D reconstruction models, and introduces the ShapeNet-VTR dataset.
Problem
Deep multi-view 3D object reconstruction models degrade sharply when test inputs come from viewpoints far from the training distribution (view transformation robustness), and existing augmentation with random views is inefficient.
Approach
Project the 3D reconstruction error voxels onto candidate viewpoints, select the views covering the most error mass, sample around them, and synthesize those views with Zero-1-to-3 to fine-tune the reconstruction model iteratively, using a category-level viewpoint pool.
Key results
- On ShapeNet-VTR Spherical with Pix2Vox++, the proposed method reaches 46.12% mIoU versus 36.71% for the Pix2Vox++ baseline and 41.62% for random-view augmentation.
- With LRGT on the Hemispherical set, the method reaches 57.11% mIoU versus 53.57% for LRGT without robustness techniques.
- The method yields only small gains on the Aligned set (e.g., 65.52% vs 64.63% mIoU for Pix2Vox++).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reconstruction error-guided view selection and synthesized views instead of the strongest CNN VTR baseline | Pix2Vox++ · ShapeNet-VTR Hemispherical | mIoU: 51.50% -> 53.53% | Helped |
| Use reconstruction error-guided view selection and synthesized views instead of the strongest CNN VTR baseline | Pix2Vox++ · ShapeNet-VTR Spherical | mIoU: 41.62% -> 46.12% | Helped |
| Use reconstruction error-guided view selection and synthesized views instead of the strongest transformer VTR baseline | LRGT · ShapeNet-VTR Hemispherical | mIoU: 56.07% -> 57.11% | Helped |
| Use reconstruction error-guided view selection and synthesized views instead of the strongest transformer VTR baseline | LRGT · ShapeNet-VTR Spherical | mIoU: 44.70% -> 44.97% | Helped |
| Set view-selection interval K to 15° instead of 30° (ablation) | Pix2Vox++ with the proposed method · ShapeNet-VTR Spherical | mIoU: 46.12% -> 43.27% | Hurt |
| Set view-selection interval K to 60° instead of 30° (ablation) | Pix2Vox++ with the proposed method · ShapeNet-VTR Spherical | mIoU: 46.12% -> 45.03% | Hurt |
| Set view-selection interval K to 90° instead of 30° (ablation) | Pix2Vox++ with the proposed method · ShapeNet-VTR Spherical | mIoU: 46.12% -> 44.26% | Hurt |
| Add the viewpoint pool (ablation) | Pix2Vox++ with the proposed method · ShapeNet-VTR Spherical | mIoU: 42.39% -> 46.12% | Helped |
Surprising findings
- Directly using Zero-1-to-3 for 3D reconstruction performs poorly, so the paper uses it only for view synthesis.
Limitations
- Gains on the Aligned in-distribution set are marginal.
- Iterative view synthesis with Zero-1-to-3 is time-consuming during training.
- Viewpoint pool may degenerate and overfit over iterations without the pool.
Experimental setup
Datasets: ShapeNet ShapeNet-VTR Pix3D SUN dataset
Benchmarks: ShapeNet-VTR Pix3D
Metrics: mIoU F-score
Baselines: Pix2Vox++ GARNet UMIFormer LRGT Nature augmentation Random augmentation VIAT MVTN OCCNet
Builds on: Pix2Vox++ LRGT Zero-1-to-3 VIAT MVTN
Closest papers by meaning
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.88)
- DMV3D: Denoising Multi-view Diffusion Using 3D Large Reconstruction Model (ICLR 2024 · similarity 0.88)
- LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias (ICLR 2025 · similarity 0.87)
- MVBoost: Boost 3D Reconstruction with Multi-View Refinement (CVPR 2025 · similarity 0.87)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.87)
- MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.