Research map / 3D vision / 3D generation
Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-to-Hero is a training-free test-time method that improves Zero-1-to-3 and Zero123-XL novel view synthesis by filtering attention maps during denoising (resampling with min-pooling and cross-step EMA), adding early-stage mutual self-attention, and using an hourglass sampling schedule.
Problem
Zero-1-to-3 novel view synthesis from a single image often produces implausible or shape-inconsistent target views, especially under large viewpoint changes, and prior fixes require retraining on more data or multi-view generation.
Approach
Treats the denoising process as an unrolled SGD-like optimization with self-attention maps as parameters, and applies attention map filtering (in-step min-pooling over resampled maps plus cross-step EMA) together with early-stage mutual self-attention from the input branch and a hourglass sampling schedule, all at test time.
Key results
- On a challenging 50-object GSO subset, Zero123-XL with Zero-to-Hero reaches PSNR 18.35, SSIM 0.864, LPIPS 0.153, IoU 78.3% vs 17.72 / 0.854 / 0.163 / 76.4% for the baseline.
- On Zero-1-to-3, Zero-to-Hero improves PSNR from 17.27 to 17.67 and IoU from 73.5% to 75.3% at 66 NFE.
- On RTMV, Zero123-XL with Zero-to-Hero improves PSNR from 10.64 to 11.15 and IoU from 71.9% to 72.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Zero-to-Hero to Zero123-XL | Zero123-XL · GSO subset | PSNR: 17.72 -> 18.35 | Helped |
| Apply Zero-to-Hero to Zero123-XL | Zero123-XL · RTMV | PSNR: 10.64 -> 11.15 | Helped |
| Add Hourglass sampling (ablation) | Zero123-XL · GSO subset | PSNR: 17.72 -> 17.74 | Helped |
| Add Hourglass sampling (ablation) | Zero-1-to-3 · GSO subset | PSNR: 17.27 -> 17.29 | Helped |
| Add resampling (ablation) | Zero123-XL with Hourglass sampling · GSO subset | PSNR: 17.74 -> 17.85 | Helped |
| Add attention map filtering (ablation) | Zero123-XL with Hourglass sampling and resampling · GSO subset | PSNR: 17.85 -> 17.92 | Helped |
| Add early-stage MSA (ablation) | Zero123-XL with Hourglass sampling and resampling · GSO subset | IoU: 76.4$\%$ -> 77.6$\%$ | Helped |
| Apply MSA throughout denoising (ablation) | Zero123-XL · GSO subset | IoU: 76.4$\%$ -> 76.6$\%$ | No clear effect |
| Apply MSA only after structure is formed (ablation) | Zero123-XL · GSO subset | IoU: 76.4$\%$ -> 76.5$\%$ | No clear effect |
| Replace Hourglass sampling with uniform DDIM sampling (ablation) | Zero123-XL · GSO subset | PSNR: 17.72 -> 17.74 | No clear effect |
| Use element-wise min-pooling instead of averaging or EMA for in-step map updates (ablation) | Zero-to-Hero attention map filtering · Novel view synthesis | performance | Helped |
| Increase resampling/filtering iterations (ablation) | Zero-to-Hero · Novel view synthesis | generation diversity | Hurt |
Surprising findings
- Simply increasing DDIM steps (25 to 100) does not improve Zero-1-to-3 or Zero123-XL metrics, and sometimes slightly degrades them.
- Cross-attention in Zero-1-to-3 degenerates to an all-ones post-softmax map, so pose conditioning acts as a global bias.
Limitations
- Limited by the generative capability of the pre-trained model; if the base model cannot generate the target pose, the method may not help.
- Filtering and resampling reduce generation diversity when applied for longer or with larger R.
- Hyperparameters were tuned on a small validation set of objects with limited computational resources.
Experimental setup
Datasets: Google Scanned Objects (GSO) RTMV
Benchmarks: Google Scanned Objects (GSO) challenging subset RTMV
Metrics: PSNR SSIM LPIPS IoU
Baselines: Zero-1-to-3 Zero123-XL
Builds on: Zero-1-to-3 MasaCtrl RePaint DDIM Stable Diffusion
Closest papers by meaning
- ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image (CVPR 2024 · similarity 0.88)
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.87)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.87)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.87)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.87)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.