Research map / Video / Video generation
Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generative Rendering renders animated untextured 3D meshes into stylized video by using a pre-trained depth-ControlNet Stable Diffusion as a multi-frame renderer, with UV-space feature blending and UV-based noise initialization for temporal consistency.
Problem
Controlling text-to-video or text-to-image diffusion models so that low-fidelity animated 3D scenes are rendered into temporally consistent, stylized animations, without a source video for inversion.
Approach
Use ground-truth 4D correspondences (UV maps) from an animated mesh to inject correspondence-aware pre- and post-attention features across frames of a depth-ControlNet T2I model, and initialize diffusion noise in a shared UV texture space instead of via DDIM inversion.
Key results
- Ours achieves frame consistency 0.9845 vs 0.9630 for Pix2Video* and 0.9822 for TokenFlow* among T2I-based methods; only Gen1 (0.9907) is higher.
- Prompt fidelity of 0.3227 for Ours, close to Per-Frame (0.3233) and above Pix2Video* (0.2983) and TokenFlow* (0.2737).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Generative Rendering instead of Per-Frame generation | StableDiffusion v1.5 · Evaluation videos | Frame Consistency: 0.9547 -> 0.9845 | Helped |
| Use Generative Rendering instead of Pix2Video* | Evaluation videos · Evaluation videos | Prompt Fidelity: 0.2983 -> 0.3227 | Helped |
| Use Generative Rendering instead of TokenFlow* | Evaluation videos · Evaluation videos | Prompt Fidelity: 0.2737 -> 0.3227 | Helped |
| Use Generative Rendering instead of Gen1 | Evaluation videos · Evaluation videos | Frame Consistency: 0.9907 -> 0.9845 | Hurt |
| Remove pre-attention feature injection (ablation) | Generative Rendering · Animation generation | visual consistency | Hurt |
| Remove post-attention feature warping (ablation) | Generative Rendering · Animation generation | pixel-wise consistency | Hurt |
| Replace UV noise initialization with fixed noise (ablation) | Generative Rendering · Animation generation | visual quality | Hurt |
Surprising findings
- Averaging post-attention features in UV space without the attention pass produces chaotic artifacts, and TokenFlow-style blending yields blurry outputs in this untextured setting.
Limitations
- Only evaluated on a small set of scenes; baselines were adapted rather than originally designed for this setting.
- Relies on 2D T2I model without temporal prior, so generation is limited by per-frame model quality and can lose fine interactions such as shadows and reflections when post-attention features are overwritten.
- Needs ground-truth UV and depth maps from an animated 3D scene as input.
Experimental setup
Datasets: Mixamo character animations physically simulated bouncing-ball scenes rendered camera-rotation scenes
Benchmarks: in-house animated 3D scene set
Metrics: frame consistency (CLIP cosine similarity) prompt fidelity (CLIP text-frame similarity)
Baselines: Pix2Video TokenFlow Gen1 Per-Frame ControlNet
Builds on: ControlNet Pix2Video TokenFlow DDIM inversion Stable Diffusion
Closest papers by meaning
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.90)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.89)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation (NeurIPS 2025 · similarity 0.89)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.