Research map / 3D vision / 3D generation
Free3D: Consistent Novel View Synthesis without 3D Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Free3D fine-tunes a pre-trained Stable Diffusion-based image generator for monocular open-set novel view synthesis using per-pixel ray conditioning (RCN), pseudo-3D cross-view attention and multi-view noise sharing, without any explicit 3D representation.
Problem
Single-image novel view synthesis that is both pose-accurate and multi-view consistent in an open-set setting, without relying on slow explicit 3D representations or extra 3D reconstruction networks.
Approach
Replace the concentrated camera-token pose encoding of Zero-1-to-3 with a per-pixel Plücker ray conditioning normalization (adaptive layer norm modulated by ray embeddings) inside a pre-trained 2D diffusion UNet, and add lightweight cross-view attention plus shared initial noise across views to improve consistency.
Key results
- On the full Objaverse test set, Free3D reaches LPIPS 0.0784 and FID 1.21 vs 0.0938 and 1.52 for Zero-1-to-3, a ~16% relative LPIPS improvement over the best baseline.
- On unseen GSO, Free3D achieves PSNR 21.13 and LPIPS 0.0619 vs Zero-1-to-3's 19.65 and 0.0758, with PPLC improving from 33.15 to 9.10 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ray embeddings to the input (ablation) | Zero-1-to-3 · Objaverse | LPIPS: 0.0938 -> 0.0858 | Helped |
| Add ray embeddings to the input (ablation) | Zero-1-to-3 · GSO | LPIPS: 0.0758 -> 0.0677 | Helped |
| Inject ray embeddings at each UNet scale (ablation) | Zero-1-to-3 with input ray embeddings · Objaverse | LPIPS: 0.0858 -> 0.0797 | Helped |
| Inject ray embeddings at each UNet scale (ablation) | Zero-1-to-3 with input ray embeddings · GSO | LPIPS: 0.0677 -> 0.0667 | Helped |
| Add ray conditioning normalization (RCN) (ablation) | Zero-1-to-3 with ray conditioning · Objaverse | LPIPS: 0.0797 -> 0.0784 | Helped |
| Add ray conditioning normalization (RCN) (ablation) | Zero-1-to-3 with ray conditioning · GSO | LPIPS: 0.0667 -> 0.0619 | Helped |
| Add pseudo-3D cross-view attention (ablation) | Free3D with RCN · Objaverse | PPLC: 15.67 -> 14.76 | Helped |
| Add pseudo-3D cross-view attention (ablation) | Free3D with RCN · GSO | PPLC: 18.48 -> 17.39 | Helped |
| Share generation noise across views (ablation) | Free3D with pseudo-3D attention · GSO | PPLC: 17.39 -> 9.10 | Helped |
| Use Free3D instead of Consistent123 | Open-set NVS · Objaverse test set | LPIPS: 0.0913 -> 0.0784 | Helped |
| Use Free3D instead of Zero123-XL | Open-set NVS · OmniObject3D | PSNR: 17.11 -> 18.23 | Helped |
| Use Free3D instead of Zero123-XL | Open-set NVS · GSO | LPIPS: 0.0706 -> 0.0619 | Helped |
Surprising findings
- Zero123-XL, trained on the much larger Objaverse-XL, does not beat Free3D on pose accuracy despite the far larger training set.
- Simple concatenation of ray embeddings alone already gives a large improvement over Zero-1-to-3, and multi-scale injection did not improve GSO generalization beyond that.
Limitations
- Pose and consistency gains depend on training only on Objaverse with the Zero-1-to-3 protocol; no training-time improvements beyond the noise schedule were explored.
- Authors note more complex designs and training strategies could further improve performance but were not the focus.
Experimental setup
Datasets: Objaverse OmniObject3D GSO
Benchmarks: Objaverse test set (7,729 objects) OmniObject3D GSO
Metrics: PSNR SSIM LPIPS FID PPLC
Baselines: Zero-1-to-3 Zero123-XL SyncDreamer Consistent123
Builds on: Zero-1-to-3 Stable Diffusion LFN 3DiM
Closest papers by meaning
- One View Is Enough: In-the-Wild Monocular Pretraining for Novel View Generation (NeurIPS 2026 · similarity 0.88)
- Unifying Correspondence Pose and NeRF for Generalized Pose-Free Novel View Synthesis (CVPR 2024 · similarity 0.88)
- ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image (CVPR 2024 · similarity 0.87)
- FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes (NeurIPS 2024 · similarity 0.87)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.87)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.