AI Paper Atlas

Research map / Video / Video generation

IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation

Yuanhao Zhai, Kevin Lin, Linjie Li, Chung-Ching Lin, Jianfeng Wang, Zhengyuan Yang, David Doermann, Junsong Yuan, Zicheng Liu, Lijuan Wang

ECCV 2024 · Video › Video generation · approach: Efficient video diffusion models · 7 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
share U-Net parameters across video and depth denoising instead of separate U-Nets (ablation)IDOL · TikTok ablation (HDNet depth)Depth L2: 0.0822 -> 0.0369Improved
add cross-modal attention between video and depth features (ablation)IDOL · TikTok ablation (HDNet depth)FID-FVD: 22.10 -> 19.28Improved
add motion consistency loss on top of cross-attention map consistency loss (ablation)IDOL · TikTok ablation (HDNet depth)FID-FVD: 19.99 -> 17.86Improved
add cross-attention map consistency loss alone (ablation)IDOL · TikTok ablation (HDNet depth)FID-FVD: 19.28 -> 19.99Mixed
use human attribute outpainting pre-training instead of DisCo HAP (ablation)IDOL · TikTok pre-training ablationFID-FVD: 19.83 -> 17.86Improved
add pre-training at all (ablation)IDOL · TikTok pre-training ablationFVD: 452.81 -> 227.86Improved
share a single cross-attention map between video and depth streams (ablation)IDOL · TikTok cross-attention map ablationDepth L2: 0.0360 -> 0.0706Hurt
generate depth jointly instead of estimating depth from DisCo generated frames with HDNetIDOL · TikTokDepth L2: 0.0975 -> 0.0336Improved
swap OpenPose/handcrafted temporal for DWPose, AnimateDiff and T2I-AdapterIDOL · TikTok (HDNet depth)FID-FVD: 17.86 -> 14.40Improved
use T2I-Adapter instead of ControlNet for pose controlIDOL · TikTok (HDNet depth)FID-FVD: 16.73 -> 14.40No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy