Research map / Image generation / Text-to-image diffusion
ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Turns a pretrained latent text-to-image diffusion U-Net into a joint multi-view generator for real-world objects by adding cross-frame-attention and volume-rendering projection layers, trained on CO3Dv2, with autoregressive generation for arbitrary viewpoints.
Problem
Generating photo-realistic, 3D-consistent multi-view images of real-world objects in authentic surroundings, since existing text-to-3D methods via SDS or synthetic-data fine-tuning yield non-photorealistic objects without backgrounds and real-world 3D datasets are small.
Approach
Augment each block of a pretrained T2I U-Net with cross-frame-attention layers (pose/intrinsics/intensity-conditioned) and a NeRF-style projection layer that unprojects features into a voxel grid and volume-renders 3D-consistent features; generate multi-view batches jointly and autoregressively condition on previously generated views.
Key results
- Unconditional generation FID improves over HoloFusion and ViewsetDiffusion, e.g., Teddybear FID 49.39 vs 81.93 (HF) and 201.71 (VD).
- Single-image reconstruction on Apple reaches PSNR 25.94 and LPIPS 0.11, versus DFM 25.79/0.07 and VD 19.54/0.31; on-par with DFM overall.
- Ablation: removing the projection layer drops PSNR from 22.24 to 16.55, while removing cross-frame attention drops it to 18.15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViewDiff instead of HoloFusion for unconditional generation | ViewDiff · Teddybear, Hydrant, Donut, and Apple | FID: 81.93 -> 49.39; 61.19 -> 46.45; 105.97 -> 68.86; 62.19 -> 56.85 | Helped |
| Add the projection layer (ablation) | ViewDiff · Teddybear and Hydrant, single-image reconstruction | PSNR: 16.55 -> 22.24 | Helped |
| Add the projection layer (ablation) | ViewDiff · Teddybear and Hydrant, single-image reconstruction | LPIPS: 0.29 -> 0.11 | Helped |
| Add the projection layer (ablation) | ViewDiff · Teddybear and Hydrant, unconditional generation | FID: 47.95 -> 47.92 | No clear effect |
| Add the projection layer (ablation) | ViewDiff · Teddybear and Hydrant, unconditional generation | KID: 0.034 -> 0.034 | No clear effect |
| Add cross-frame attention (ablation) | ViewDiff · Teddybear and Hydrant, single-image reconstruction | PSNR: 18.15 -> 22.24 | Helped |
| Add cross-frame attention (ablation) | ViewDiff · Teddybear and Hydrant, single-image reconstruction | LPIPS: 0.25 -> 0.11 | Helped |
| Add cross-frame attention (ablation) | ViewDiff · Teddybear and Hydrant, unconditional generation | FID: 47.93 -> 47.92 | No clear effect |
| Add cross-frame attention (ablation) | ViewDiff · Teddybear and Hydrant, unconditional generation | KID: 0.034 -> 0.034 | No clear effect |
| Use ViewDiff instead of DFM | ViewDiff · Donut, single-image reconstruction | PSNR: 23.91 -> 21.50 | Hurt |
| Use ViewDiff instead of DFM | ViewDiff · Hydrant, single-image reconstruction | PSNR: 22.67 -> 22.49 | No clear effect |
Limitations
- Occasional slight view inconsistency due to view-dependent exposure effects in the real-world training data.
- Focuses on object-centric generation; scene-scale generation not explored.
- Single-image reconstruction on some categories (e.g., Donut LPIPS 0.18) trails DFM.
Experimental setup
Datasets: CO3Dv2 BLIP-2 generated captions
Benchmarks: CO3Dv2
Metrics: FID KID PSNR SSIM LPIPS
Baselines: HoloFusion ViewsetDiffusion DFM DreamFusion Zero-1-to-3
Builds on: latent diffusion Zero-1-to-3 IBRNet MERF LoRA
Closest papers by meaning
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.91)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.90)
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.90)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.90)
- Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data (CVPR 2025 · similarity 0.90)
- IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.