Research map / Video / Video generation
Unified Dense Prediction of Video Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds UDPDiff, a unified CogVideoX-5B-based video diffusion model that jointly generates RGB video with entity segmentation and depth colormaps (Pixelplanes) under learned task embeddings, and introduces the Panda-Dense dataset (~300K captioned videos with segmentation and depth).
Problem
Video diffusion models produce temporally inconsistent and unrealistic motion, and no large video dataset jointly provides captions, videos, entity segmentation, and depth maps for dense-prediction-aware video generation.
Approach
Encode entity masks and depth maps as RGB colormaps (Pixelplanes, with random per-entity colors and spectral depth projection) and jointly denoise them with the video latent via channel concatenation, selecting the dense task with a learned task embedding added to the timestep embedding.
Key results
- Single-task UDPDiff (segmentation) raises motion smoothness to 98.62 vs 97.67 for CogVideoX 5B on VBench-based metrics.
- Multi-task UDPDiff with depth reduces FVD from 343.92 (CogVideoX 5B) to 302.55 and improves motion smoothness to 99.23.
- Inference time is 205.81 s for multi-task vs 204.46 s for CogVideoX 5B, indicating negligible overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add segmentation guidance to video generation | CogVideoX 5B · Panda-70M evaluation | FVD: 343.92 -> 316.76 | Helped |
| Add depth-estimation guidance to video generation | CogVideoX 5B · Panda-70M evaluation | FVD: 343.92 -> 302.55 | Helped |
| Add segmentation guidance to video generation | CogVideoX 5B · Panda-70M evaluation | SC: 94.57 -> 95.21 | Mixed |
| Add depth-estimation guidance to video generation | CogVideoX 5B · Panda-70M evaluation | RMSE: 0.0907 -> 0.1634 | Hurt |
| Use learned task embeddings rather than text prompts to distinguish tasks (ablation) | UDPDiff multi-task · Panda-70M evaluation, depth inference | FVD: 321.43 -> 302.55 | Helped |
| Replace location-aware colormap with Pixelplanes (ablation) | UDPDiff · Panda-70M evaluation | SC: 81.26 -> 94.98 | Mixed |
| Use equal segmentation and depth data partition (0.5, 0.5) (ablation) | UDPDiff multi-task · Panda-70M evaluation | MS: 98.16 -> 99.23; 96.97 -> 99.23 | Helped |
| Generate depth training data with DepthCrafter instead of Depth Anything V2 (ablation) | Panda-Dense pipeline · Panda-70M evaluation | MS: 96.13 -> 98.10 | Helped |
| Initialize SAM 2 with EntitySeg rather than a point grid (ablation) | Panda-Dense pipeline · Panda-70M evaluation | MS: 96.18 -> 98.62; 96.35 -> 98.62 | Helped |
Surprising findings
- The location-aware colormap from UniGS performs far worse than Pixelplanes on consistency metrics (SC 81.26 vs 94.98).
Limitations
- Depth accuracy evaluated against pseudo-ground-truth from DepthCrafter, not real ground truth.
- Compared only against CogVideoX 5B and fine-tuned under limited compute.
Experimental setup
Datasets: Panda-70M Panda-Dense
Benchmarks: VBench FVD
Metrics: subject consistency background consistency motion smoothness FVD delta accuracy RMSE
Baselines: CogVideoX 5B UniGS location-aware colormap Depth Anything V2
Builds on: CogVideoX UniGS Emu Edit Panda-70M DepthCrafter
Closest papers by meaning
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.86)
- Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes (CVPR 2024 · similarity 0.85)
- FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.85)
- DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models (NeurIPS 2025 · similarity 0.85)
- Exploiting Diffusion Prior for Generalizable Dense Prediction (CVPR 2024 · similarity 0.85)
- Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.