Research map / Video / Video generation
Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Through-The-Mask, a two-stage I2V framework that first generates mask-based motion trajectories and then produces video via masked cross- and self-attention, and introduces the SA-V-128 benchmark.
Problem
Image-to-video generation often produces inconsistent or unfaithful object motion, especially with multiple objects, because end-to-end models must implicitly reason about semantics and motion together.
Approach
Use a time-consistent per-frame object segmentation mask trajectory (rather than optical flow) as an intermediate representation, and inject it into the video diffusion model via per-object masked cross-attention (object prompts to regions) and masked spatio-temporal self-attention (attention restricted to same object).
Key results
- On SA-V-128 single-object (U-Net), FVD drops from 1285.99 (TI2V) to 925.39 and ViCLIP-V rises to 0.888.
- On Image-Animation-Bench (U-Net), FVD is 196.23 vs 242.18 for TI2V and 217.40 for DynamiCrafter.
- Flow-based trajectories yield FVD 1014.72 vs 648.59 for mask-based in the U-Net ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mask-based motion trajectories and masked attention rather than the TI2V single-stage baseline | I2V generation with UNet · SA-V-128 single-object | FVD: 1285.99 -> 925.39 | Helped |
| Use mask-based motion trajectories and masked attention rather than the TI2V single-stage baseline | I2V generation with UNet · SA-V-128 multi-object | FVD: 1410.68 -> 1089.86 | Helped |
| Use the proposed method rather than the strongest listed baseline | I2V generation · Image-Animation-Bench | FVD: 217.40 -> 196.23 | Helped |
| Add masked attention (ablation) | I2V generation with UNet · SA-V-128 ablation | FVD: 972.25 -> 648.59 | Helped |
| Add masked attention (ablation) | I2V generation with DiT · SA-V-128 ablation | FVD: 1182.49 -> 1082.23 | Helped |
| Add masked self-attention to masked cross-attention (ablation) | I2V generation with UNet · SA-V-128 ablation | FVD: 670.92 -> 648.59 | Helped |
| Add masked cross-attention to masked self-attention (ablation) | I2V generation with UNet · SA-V-128 ablation | FVD: 658.92 -> 648.59 | Helped |
| Add masked self-attention to masked cross-attention (ablation) | I2V generation with DiT · SA-V-128 ablation | FVD: 1105.91 -> 1082.23 | Helped |
| Add masked cross-attention to masked self-attention (ablation) | I2V generation with DiT · SA-V-128 ablation | FVD: 1152.38 -> 1082.23 | Helped |
| Replace optical-flow motion trajectories with segmentation-mask trajectories (ablation) | I2V generation with UNet · SA-V-128 motion representation ablation | FVD: 1014.72 -> 648.59 | Helped |
| Use mask-based motion trajectories and masked attention rather than TI2V | I2V generation with UNet · SA-V-128 single-object | ViCLIP-T: 0.219 -> 0.220 | No clear effect |
Surprising findings
- In human evaluation on Image-Animation-Bench, TI2V slightly beat the proposed method on text alignment (50.8 vs 49.2).
Limitations
- Relies on an external pipeline (LLM prompt extraction, Grounding DINO, SAM2) to produce training trajectories and prompts.
- Human preference is mixed on text alignment versus the TI2V baseline.
- Motion-capable objects are limited to those the LLM extracts from the text.
Experimental setup
Datasets: SA-V Image-Animation-Bench
Benchmarks: SA-V-128 Image-Animation-Bench
Metrics: FVD CLIPFrame ViCLIP-T ViCLIP-V Average Displacement human preference
Baselines: VideoCrafter DynamiCrafter Motion-I2V ConsistI2V TI2V
Builds on: Motion-I2V Masked cross-attention (Multi-object text-to-image, Couairon/Lee et al. style ob AnimateDiff Movie Gen LDM
Closest papers by meaning
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.89)
- Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation (CVPR 2025 · similarity 0.88)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.88)
- MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformer (ICLR 2025 · similarity 0.87)
- TrajLoc: Trajectory-Attention Localization for Multi-Object Motion Control (NeurIPS 2026 · similarity 0.87)
- Decouple Content and Motion for Conditional Image-to-Video Generation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.