Research map / Video / Video generation
ActAnywhere: Subject-Aware Video Background Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ActAnywhere, a latent video diffusion model that generates a video background conditioned on a foreground subject segmentation sequence and a single background/composited condition frame.
Problem
Automatically generating video background that adapts to foreground subject motion and follows an artist-provided condition frame, a task traditionally done with tedious manual VFX work.
Approach
Condition a latent video diffusion model (AnimateDiff-based 3D U-Net) on the VAE-encoded foreground segmentation sequence, its masks, and CLIP features of a single background/composited condition frame, trained self-supervised on frames from human-scene interaction videos.
Key results
- Model generates temporally coherent, realistic foreground-background interactions and significantly outperforms baselines (Gen1, TokenFlow, Control-A-Video, AnimateDiff, VideoCrafter1) qualitatively on DAVIS and HiC+.
- Generalizes zero-shot to non-human subjects despite training only on human-scene videos.
- Generates one video in about 8.5 seconds on a single NVIDIA A100.
Surprising findings
- Certain general video inpainting and outpainting capability emerges from the model when the condition is disabled, possibly due to random condition dropping.
Limitations
- Failure cases when segmentation excludes parts of the subject (e.g., dress texture) or the condition frame has errors (e.g., broomstick direction).
- Inherits demographic biases from the training data.
- Evaluation is primarily qualitative; no quantitative metrics reported in the text.
Experimental setup
Datasets: HiC+ DAVIS HVU Charades Moments WebVid10M
Benchmarks: DAVIS HiC+ held-out set
Baselines: Gen1 Text2LIVE TokenFlow Control-A-Video AnimateDiff VideoCrafter1
Builds on: AnimateDiff Stable Diffusion inpainting Latent Video Diffusion CLIP
Closest papers by meaning
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.85)
- Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation (NeurIPS 2026 · similarity 0.85)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.85)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.85)
- ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation (NeurIPS 2024 · similarity 0.84)
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.