Research map / Video / Video generation
IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation
Yuanhao Zhai, Kevin Lin, Linjie Li, Chung-Ching Lin, Jianfeng Wang, Zhengyuan Yang, David Doermann, Junsong Yuan, Zicheng Liu, Lijuan Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| share U-Net parameters across video and depth denoising instead of separate U-Nets (ablation) | IDOL · TikTok ablation (HDNet depth) | Depth L2: 0.0822 -> 0.0369 | Improved |
| add cross-modal attention between video and depth features (ablation) | IDOL · TikTok ablation (HDNet depth) | FID-FVD: 22.10 -> 19.28 | Improved |
| add motion consistency loss on top of cross-attention map consistency loss (ablation) | IDOL · TikTok ablation (HDNet depth) | FID-FVD: 19.99 -> 17.86 | Improved |
| add cross-attention map consistency loss alone (ablation) | IDOL · TikTok ablation (HDNet depth) | FID-FVD: 19.28 -> 19.99 | Mixed |
| use human attribute outpainting pre-training instead of DisCo HAP (ablation) | IDOL · TikTok pre-training ablation | FID-FVD: 19.83 -> 17.86 | Improved |
| add pre-training at all (ablation) | IDOL · TikTok pre-training ablation | FVD: 452.81 -> 227.86 | Improved |
| share a single cross-attention map between video and depth streams (ablation) | IDOL · TikTok cross-attention map ablation | Depth L2: 0.0360 -> 0.0706 | Hurt |
| generate depth jointly instead of estimating depth from DisCo generated frames with HDNet | IDOL · TikTok | Depth L2: 0.0975 -> 0.0336 | Improved |
| swap OpenPose/handcrafted temporal for DWPose, AnimateDiff and T2I-Adapter | IDOL · TikTok (HDNet depth) | FID-FVD: 17.86 -> 14.40 | Improved |
| use T2I-Adapter instead of ControlNet for pose control | IDOL · TikTok (HDNet depth) | FID-FVD: 16.73 -> 14.40 | No clear effect |
Closest papers by meaning
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.87)
- Learning Temporally Consistent Video Depth from Video Diffusion Priors (CVPR 2025 · similarity 0.86)
- Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusion (AAAI 2025 · similarity 0.86)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.86)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.86)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
