Research map / 3D vision / NeRF and radiance fields
Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Efficient-3DiM speeds up training of a Zero-1-to-3-style single-image novel-view diffusion synthesizer from 10 days to under 1 day on 8 A100s via Gaussian timestep sampling, DINOv2 multi-scale feature conditioning, and mixed precision/caching/cosine LR training.
Problem
Training diffusion models for single-image novel view synthesis (e.g., Zero-1-to-3) requires very long training (about 10 days on 8 A100 GPUs), limiting research iteration.
Approach
Replace uniform diffusion timestep sampling with a biased Gaussian sampling concentrated on high-noise timesteps (where view/geometry is formed), swap the CLIP image encoder for DINOv2 features amalgamated multi-scale into the UNet, and add engineering tweaks (mixed precision with extra LayerNorm, cached DINOv2 features, cosine LR) to cut training time without extra compute.
Key results
- Full Efficient-3DiM reaches 0.171 LPIPS and 0.128 MSE on the Objaverse validation set after 20k steps, versus 0.236/0.187 for the no-component baseline.
- Training time is reduced from 10 days to under 1 day on 8 A100 GPUs, reported as up to 14x speedup over Zero 1-to-3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Gaussian timestep sampling (mean = 1000, std=200) (ablation) | Efficient-3DiM · Objaverse validation | LPIPS: 0.236 -> 0.205 | Helped |
| Add Gaussian timestep sampling (mean = 1000, std=200) (ablation) | Efficient-3DiM · Objaverse validation | MSE: 0.187 -> 0.142 | Helped |
| Replace CLIP with DINO-v2 features (ablation) | Efficient-3DiM with Gaussian sampling · Objaverse validation | LPIPS: 0.205 -> 0.187 | Helped |
| Replace CLIP with DINO-v2 features (ablation) | Efficient-3DiM with Gaussian sampling · Objaverse validation | MSE: 0.142 -> 0.136 | Helped |
| Add spatial feature amalgamation to the UNet (ablation) | Efficient-3DiM with Gaussian sampling and CLIP · Objaverse validation | LPIPS: 0.205 -> 0.191 | Helped |
| Add spatial feature amalgamation to the UNet (ablation) | Efficient-3DiM with Gaussian sampling and CLIP · Objaverse validation | MSE: 0.142 -> 0.144 | Helped |
| Add spatial feature amalgamation to the UNet (ablation) | Efficient-3DiM with Gaussian sampling and DINO-v2 · Objaverse validation | LPIPS: 0.187 -> 0.171 | Helped |
| Add spatial feature amalgamation to the UNet (ablation) | Efficient-3DiM with Gaussian sampling and DINO-v2 · Objaverse validation | MSE: 0.136 -> 0.128 | Helped |
| Use Gaussian sampling with mean = 1000 and std=100, leaving early diffusion steps almost unupdated (ablation) | Efficient-3DiM · Objaverse validation | LPIPS | Hurt |
| Use mixed-precision training instead of full-precision training (ablation) | Efficient-3DiM · Training time | training time: 40% | Helped |
| Add Layer Normalization before passing DINO-v2 features to the diffusion model (ablation) | Efficient-3DiM with mixed precision · Training stability | stability | Helped |
| Cache DINO-v2 outputs on disk (ablation) | Efficient-3DiM · Training time | training time: 10% | Helped |
Surprising findings
- Gaussian timestep sampling centered at t=1000 showed that early (low-noise) timesteps barely need updates, with the novel-view ability emerging mainly from late, noisy stages.
Limitations
- Multi-view consistency is not fully addressed and is left for future work.
- Ablations use only 20k training steps and LPIPS/MSE metrics, with no large-scale quantitative comparison to other methods.
Experimental setup
Datasets: Objaverse
Benchmarks: Objaverse validation set
Metrics: LPIPS MSE
Baselines: Zero 1-to-3 CLIP-conditioned Zero-1-to-3 uniform timestep sampling
Builds on: Zero-1-to-3 Stable Diffusion DDIM 3DiM
Closest papers by meaning
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.91)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.91)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.90)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.90)
- NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer (ICLR 2025 · similarity 0.89)
- WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.