Research map / 3D vision / 3D generation
Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Repurposes Stable Video Diffusion as a joint neural inverse renderer (G-buffer estimation from video) and forward renderer (photorealistic video from G-buffers and HDR lighting), trained on synthetic data plus auto-labeled real videos.
Problem
Classic physically-based rendering and inverse rendering need precise geometry, materials and lighting that are hard to obtain for real-world videos, and prior neural methods are image-only, data-hungry, or weak on real-world forward rendering.
Approach
Fine-tune a video diffusion model as an inverse renderer on synthetic data, use it to auto-label real videos with G-buffers, then co-train a forward renderer on synthetic and auto-labeled real data with a real-data LoRA and a multi-level environment-map encoder injected via cross-attention.
Key results
- Forward rendering on SyntheticScenes: PSNR 26.0 / LPIPS 0.201, versus 24.8 for SSRT and 18.5 for RGB↔X.
- Relighting on SyntheticObjects: PSNR 27.50 / LPIPS 0.067, versus 26.39 for Neural Gaffer and 23.79 for DiLightNet.
- Inverse rendering on SyntheticScenes: albedo si-PSNR 26.7 and normals angular error 5.97°, versus RGB↔X at 19.6 and 23.80°.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a video model instead of per-frame image inference (ablation) | Diffusion Renderer forward rendering · SyntheticObjects and SyntheticScenes | PSNR, SSIM, and LPIPS: 27.4 -> 28.3 PSNR; 25.4 -> 26.0 PSNR | Helped |
| use a dedicated environment map encoder instead of concatenating encoded environment maps to image channels (ablation) | Diffusion Renderer forward rendering · SyntheticObjects and SyntheticScenes | PSNR, SSIM, and LPIPS: 27.8 -> 28.3 PSNR; 25.3 -> 26.0 PSNR | Helped |
| add split-sum shading buffers as conditioning (ablation) | Diffusion Renderer forward rendering · SyntheticObjects and SyntheticScenes | PSNR, SSIM, and LPIPS: + shading cond.: 28.7 PSNR, 0.930 SSIM, 0.056 LPIPS; 25.6 PSNR, 0.761 SSIM, 0.245 LPIPS | No clear effect |
| use video inference instead of image inference (ablation) | Diffusion Renderer inverse rendering · SyntheticScenes | Metallic RMSE: 0.066 to 0.039 | Helped |
| use video inference instead of image inference (ablation) | Diffusion Renderer inverse rendering · SyntheticScenes | Roughness RMSE: 0.098 to 0.078 | Helped |
| fine-tune as a 1-step deterministic model instead of multi-step stochastic inference (ablation) | Diffusion Renderer inverse rendering · SyntheticScenes and InteriorVerse | PSNR and LPIPS: PSNR: 25.0 -> 26.0; LPIPS: 0.205 -> 0.219 | Mixed |
| train on synthetic and real-world data instead of synthetic data alone (ablation) | Diffusion Renderer relighting · Real-world scenes | visual quality | Helped |
| add LoRA during training with real-world data (ablation) | Diffusion Renderer relighting · Real-world scenes | visual quality | Helped |
Surprising findings
- One-step deterministic fine-tuning gave higher PSNR than the multi-step stochastic inverse renderer, though it had worse LPIPS due to blurrier outputs.
- Synthetic-only training of the forward renderer struggled with real-world structures such as trees, so real auto-labeled data was needed.
Limitations
- Based on Stable Video Diffusion, which is offline and slow at inference.
- Editing may introduce slight color or texture variations relative to the original content.
- Real-world auto-labeling depends on an off-the-shelf lighting estimator (DiffusionLight) that may be inaccurate.
Experimental setup
Datasets: Objaverse LVIS PolyHaven DL3DV10k InteriorVerse HyperSim
Benchmarks: SyntheticScenes SyntheticObjects InteriorVerse
Metrics: PSNR SSIM LPIPS si-PSNR si-LPIPS RMSE mean angular error
Baselines: SSRT SplitSum RGB↔X DiLightNet Neural Gaffer Kocsis et al. IIW Li et al. Zhu et al.
Builds on: Stable Video Diffusion RGB↔X Neural Gaffer DiffusionLight EDM
Closest papers by meaning
- Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusion (CVPR 2025 · similarity 0.93)
- Channel-wise Noise Scheduled Diffusion for Inverse Rendering in Indoor Scenes (CVPR 2025 · similarity 0.90)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.89)
- BLURD: Benchmarking and Learning using a Unified Rendering and Diffusion Model (NeurIPS 2024 · similarity 0.89)
- Neural Gaffer: Relighting Any Object via Diffusion (NeurIPS 2024 · similarity 0.89)
- DORSal: Diffusion for Object-centric Representations of Scenes $\textit{et al.}$ (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.