Research map / 3D vision / 3D generation
ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConsistNet, a lightweight plug-in multi-view consistency block that unprojects per-view U-Net features into a shared 3D volume and reprojects them back, enforcing 3D consistency over a frozen Zero123-XL for single-image multi-view generation.
Problem
Single-image novel-view generation with latent diffusion models (e.g., Zero123-XL) produces views that are semantically plausible but do not adhere to a shared 3D structure, lacking multi-view geometric consistency.
Approach
Run parallel per-view diffusion models and insert a ConsistNet block into each U-Net decoder layer that unprojects features into a world-coordinate 3D volume, applies cross-view self-attention and 3D convolutions, then ray-samples and cross-attends back to each view, with zero-initialized residual outputs; only the block is trained while the backbone stays frozen.
Key results
- At elevation 0, ConsistNet reaches LPIPS Alex 0.15, SSIM 0.89, PSNR 22.75 on Google Scanned Objects versus 0.19 / 0.85 / 18.25 for Zero123-XL, in 11 s versus 5 s baseline runtime.
- At elevation 30, ConsistNet performs comparably to SyncDreamer (PSNR 23.67 vs 23.81) while running about 10x faster (11 s vs 2 min).
- Plugging ConsistNet into the decoder only gives the best quality-efficiency trade-off; on Zero123 base PSNR is 21.88 versus 23.67 on Zero123-XL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ConsistNet to Zero123-XL | Zero123-XL · Google Scanned Objects, elevations 0 and 15 | LPIPS Alex: 0.19 -> 0.15; 0.23 -> 0.12 | Helped |
| add ConsistNet to Zero123-XL | Zero123-XL · Google Scanned Objects, elevation 30 | LPIPS Alex: 0.27 -> 0.11 | Hurt |
| add ConsistNet to Zero123-XL | Zero123-XL · Google Scanned Objects, elevation 30 | SSIM: 0.90 -> 0.90 | No clear effect |
| plug ConsistNet into decoder only instead of encoder only (ablation) | ConsistNet · Google Scanned Objects | LPIPS Alex: 0.14 -> 0.11 | Helped |
| plug ConsistNet into encoder and decoder (ablation) | ConsistNet · Google Scanned Objects | LPIPS Alex: 0.11 -> 0.11 | No clear effect |
| plug ConsistNet before each U-Net block rather than after (ablation) | ConsistNet · Google Scanned Objects | LPIPS Alex: 0.11 -> 0.12 | Hurt |
| replace Zero123-XL backbone with Zero123 (ablation) | ConsistNet · Google Scanned Objects | LPIPS Alex: 0.11 -> 0.14 | Hurt |
| reduce generation to 4 views (ablation) | ConsistNet + Zero123-XL · Google Scanned Objects | LPIPS Alex: 0.11 -> 0.13 | Hurt |
| reduce generation to 8 views (ablation) | ConsistNet + Zero123-XL · Google Scanned Objects | LPIPS Alex: 0.11 -> 0.11 | No clear effect |
| increase denoising steps from 50 to 100 (ablation) | ConsistNet + Zero123-XL · Google Scanned Objects | LPIPS Alex: 0.11 -> 0.11 | No clear effect |
| increase denoising steps from 50 to 200 (ablation) | ConsistNet + Zero123-XL · Google Scanned Objects | LPIPS Alex: 0.11 -> 0.11 | No clear effect |
Surprising findings
- The ConsistNet block is trainable only as a small residual MLP plus aggregation networks, with the Zero123-XL backbone frozen, yet it improves over the base model at all elevations.
Limitations
- Quantitative evaluation against a single ground-truth view is ill-suited because single-image novel view generation is ambiguous.
- Performance at 30-degree elevation is only comparable to SyncDreamer rather than better.
- Evaluated on a single dataset (Google Scanned Objects) for quantitative results.
Experimental setup
Datasets: Objaverse Google Scanned Objects
Benchmarks: Google Scanned Objects
Metrics: LPIPS Alex LPIPS VGG SSIM MS-SSIM PSNR runtime
Baselines: Zero123-XL DreamFusion SyncDreamer Zero123
Builds on: Zero123-XL ControlNet Latent Diffusion Models
Closest papers by meaning
- MET3R: Measuring Multi-View Consistency in Generated Images (CVPR 2025 · similarity 0.89)
- ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing (CVPR 2024 · similarity 0.88)
- HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D (CVPR 2024 · similarity 0.88)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.88)
- SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (ICLR 2024 · similarity 0.87)
- MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.