Research map / 3D vision / 3D generation
WildCAT3D: Appearance-Aware Multi-View Diffusion in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WildCAT3D, an appearance-aware multi-view diffusion model (CAT3D extension) that learns scene-level novel view synthesis from in-the-wild photos using a generalizable appearance encoder and warp conditioning.
Problem
Scene-level sparse novel view synthesis is limited by scarce clean multi-view training data; abundant permissively licensed in-the-wild photos have varying appearance (lighting, weather, aspect ratio) that breaks standard multi-view diffusion training.
Approach
Explicitly model global appearance with a feed-forward appearance encoder whose low-dimensional embedding conditions the multi-view diffusion model (copied from the source view at inference), use an appearance-aware classifier-free guidance to avoid oversaturation, and add warp conditioning from depth-based warps of the first view to resolve scale ambiguity.
Key results
- On DTU single-view NVS, WildCAT3D reaches PSNR 10.77, LPIPS 0.388, FID 57.32, outperforming the MegaScenes NVS baseline (PSNR 8.795, FID 85.96).
- On Mip-NeRF 360, WildCAT3D achieves PSNR 14.77 and FID 42.17 versus 14.06 and 64.41 for MS NVS.
- On MegaScenes, quantitative results are comparable to the MS NVS baseline (PSNR 13.92 vs 13.40), while qualitative gains are larger on novel trajectories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add warp conditioning (ablation) | WildCAT3D · DTU | PSNR: 9.795 -> 10.77 | Helped |
| add warp conditioning (ablation) | WildCAT3D · Mip-NeRF 360 | PSNR: 13.98 -> 14.77 | Helped |
| add warp conditioning (ablation) | WildCAT3D · Re10K | PSNR: 18.97 -> 21.58 | Helped |
| add warp conditioning (ablation) | WildCAT3D · MegaScenes | PSNR: 13.45 -> 13.92 | Helped |
| add warp conditioning and appearance modeling (ablation) | CAT3D fine-tuned on in-the-wild data · DTU | PSNR: 8.699 -> 10.77 | Helped |
| add warp conditioning and appearance modeling (ablation) | CAT3D fine-tuned on in-the-wild data · Mip-NeRF 360 | PSNR: 13.90 -> 14.77 | Helped |
| add warp conditioning and appearance modeling (ablation) | CAT3D fine-tuned on in-the-wild data · Re10K | PSNR: 17.73 -> 21.58 | Helped |
| add warp conditioning and appearance modeling (ablation) | CAT3D fine-tuned on in-the-wild data · MegaScenes | PSNR: 12.63 -> 13.92 | Helped |
| fine-tune base CAT3D on in-the-wild data with WildCAT3D components (ablation) | Base CAT3D · Re10K | PSNR: 21.89 -> 21.58 | Hurt |
| fine-tune base CAT3D on in-the-wild data with WildCAT3D components (ablation) | Base CAT3D · DTU | PSNR: 10.25 -> 10.77 | Helped |
| fine-tune base CAT3D on in-the-wild data with WildCAT3D components (ablation) | Base CAT3D · Mip-NeRF 360 | PSNR: 13.92 -> 14.77 | Helped |
| fine-tune base CAT3D on in-the-wild data with WildCAT3D components (ablation) | Base CAT3D · MegaScenes | PSNR: 12.91 -> 13.92 | Helped |
Surprising findings
- Naively applying classifier-free guidance to appearance conditions causes oversaturation, motivating the appearance-aware CFG variant.
- Base CAT3D without in-the-wild fine-tuning outperforms WildCAT3D on Re10K, which the authors attribute to overfitting to an in-distribution dataset.
Limitations
- Generated views are not guaranteed to be fully 3D consistent; mild flickering and saturation changes persist.
- Relies on an existing monocular depth estimation pipeline whose errors propagate to novel views.
- Text control is only coarse, via CLIP retrieval over about 10K MegaScenes images, with no explicit textual conditioning.
Experimental setup
Datasets: MegaScenes CO3D RealEstate10K
Benchmarks: DTU Mip-NeRF 360 Re10K MegaScenes
Metrics: PSNR SSIM LPIPS FID KID
Baselines: Zero-1-to-3 ZeroNVS SD-Inpaint MegaScenes NVS
Builds on: CAT3D Latent Diffusion Model ZeroNVS Zero-1-to-3 classifier-free guidance
Closest papers by meaning
- One View Is Enough: In-the-Wild Monocular Pretraining for Novel View Generation (NeurIPS 2026 · similarity 0.89)
- Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections (NeurIPS 2024 · similarity 0.89)
- ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image (CVPR 2024 · similarity 0.88)
- MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model (CVPR 2025 · similarity 0.88)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.88)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.