AI Paper Atlas

Research map / Video / Video generation

SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models

Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, Bo Dai

ECCV 2024 · Video › Video generation · approach: Compositional text-to-video diffusion · 152 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use SparseCtrl sparse encoder instead of frame-wise ControlNet as control becomes sparserAnimateDiff · Sparse depth-to-video, r_mask=7/8depth MAE err. (x100): 14.84 -> 7.40Improved
use SparseCtrl in dense setting (r_mask=0) compared with dense-control baselinesAnimateDiff · Sparse depth-to-video, r_mask=0depth MAE err. (x100): 8.24 -> 8.92Hurt
add temporal propagation layers to the frame-wise condition encoder (ablation)SparseCtrl encoder · image animation (qualitative)temporal consistencyImproved
remove noised sample input from the condition encoder (ablation)SparseCtrl encoder with propagation layers · image animation (qualitative)visual qualityImproved
use SparseCtrl instead of VideoComposer for image animationimage animation · Image animation (8 images)LPIPS first-frame: 0.3346 -> 0.1467Improved
use SparseCtrl instead of DynamiCrafter for image animationimage animation · Image animation (8 images)CLIP temporal consistency: 98.49 -> 95.25Hurt
use incomplete or uninformative text prompts with sketch condition (ablation)SparseCtrl sketch-to-video · sketch-to-video (qualitative)content generationHurt

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 2 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy