Research map / Video / Video generation
ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a pre-trained DiT text-to-video model with learnable transition tokens and local attention masking, trained on multi-shot videos built from single-shot datasets, to generate controllable text-to-multi-shot videos.
Problem
Existing diffusion text-to-video models generate only single continuous shots and cannot produce multi-shot videos with hard cuts where the same character performs different activities across same or different backgrounds.
Approach
A model-agnostic adapter-style fine-tuning that adds a learnable transition token per shot boundary and a local attention mask so each shot's text tokens attend only to that shot's visual tokens, enabling shot-level prompts, counts, and durations within a single full-attention video.
Key results
- ShotAdapter achieves identity consistency of 78.67 / 86.33 / 70.30 (diff bg, same bg, 2-shot) and higher IC than baselines for 3- and 4-shot videos, per Table 1.
- Background consistency for same-background samples improves to 76.44 / 61.86 / 74.89 across 2/3/4 shots versus 66-73 for the best baseline (SR), per Table 1.
- Transition token yields mean shot duration error of 0.83 to 2.00 frames per shot across 2 to 8 shot videos (Table 2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add transition tokens (ablation) | ShotAdapter · 2-, 3-, and 4-shot videos | Identity Consistency: 77.17 -> 78.67; 84.78 -> 86.33; 68.95 -> 70.30 | Helped |
| fine-tune on the full multi-shot dataset instead of only 2-shot videos (ablation) | ShotAdapter · 3- and 4-shot videos | Identity Consistency: 85.46 -> 86.33; 70.12 -> 70.30 | Mixed |
| use ShotAdapter rather than Similar Shots | multi-shot video generation · 3-shot, diff bg | Identity Consistency: 82.55 -> 86.33 | Helped |
| use ShotAdapter rather than Shots by Reference | multi-shot video generation · same bg | Background Consistency: 82.11 -> 89.48 | Helped |
Surprising findings
- Fine-tuning on only 2-shot videos still generalizes to 3- and 4-shot videos with better identity and background consistency than baselines.
- In the user study, the method slightly trails baselines in some text-alignment comparisons, which the authors attribute to smaller batch size and baselines ignoring parts of prompts.
Limitations
- Limited to human foreground subjects; non-human subjects were not tested.
- Maximum generation length limited to 128 frames by the base model.
- Small user-study quality drop relative to some baselines; attributed partly to smaller batch size.
Experimental setup
Datasets: single-shot video dataset (unnamed) RAFT-based large-motion filtered videos
Benchmarks: ShotAdapter multi-shot evaluation set (48 prompts, 2/3/4 shots)
Metrics: Identity Consistency (DINOv2) Background Consistency (DINOv2) Text Alignment (ViCLIP) User study preference Mean Shot Duration Error (MSDE)
Baselines: Random Shots Similar Shots Shots by Reference (StoryMaker + I2V) single-shot generation SEINE MEVG FreeNoise Gen-L-Video
Builds on: OpenSora MovieGen joint attention (MMDiT) DreamBooth
Closest papers by meaning
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.90)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.89)
- VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · similarity 0.89)
- EchoShot: Multi-Shot Portrait Video Generation (NeurIPS 2025 · similarity 0.88)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.