Research map / Video / Video generation
SimDA: Simple Diffusion Adapter for Efficient Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a frozen Stable Diffusion T2I model to text-to-video generation, editing and 1024x1024 super-resolution by training only ~24M adapter parameters plus a latent-shift attention module.
Problem
Text-to-video diffusion models are expensive to train from scratch or by fully fine-tuning large T2I models (billions of parameters), limiting efficient video generation.
Approach
Freeze a pretrained Stable Diffusion U-Net and insert lightweight spatial adapters (attention and FFN bottleneck adapters), a depth-wise 3D-conv temporal adapter, and a latent-shift attention that concatenates tokens from the previous frames as keys/values for temporal consistency without new parameters.
Key results
- On MSR-VTT zero-shot, SimDA reaches FVD 456 and CLIPSIM 0.2945 with 1.08B total parameters, while fine-tuning only about 24M parameters.
- On the WebVid validation set, SimDA achieves FVD 363.98 and CLIPSIM 0.3054, beating ModelScope and LVDM and competitive with VideoFactory (292.35, 0.3070).
- Ablation: removing the latent-shift attention raises FVD from 530.2 to 618.2 and lowers CLIPSIM from 0.3034 to 0.3011 on 1K WebVid samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the Temporal Adapter (ablation) | SimDA · WebVid validation (1K samples) | FVD: 1470.1 -> 530.2 | Helped |
| add the Spatial Adapter (ablation) | SimDA · WebVid validation (1K samples) | FVD: 811.3 -> 530.2 | Helped |
| add the Attention Adapter (ablation) | SimDA · WebVid validation (1K samples) | FVD: 764.8 -> 530.2 | Helped |
| add the FFN Adapter (ablation) | SimDA · WebVid validation (1K samples) | FVD: 623.7 -> 530.2 | Helped |
| use Latent-Shift Attention instead of regular single-frame spatial attention (ablation) | SimDA · WebVid validation (1K samples) | FVD: 618.2 -> 530.2 | Helped |
| use SimDA instead of LVDM | text-to-video generation · MSR-VTT | FVD: 742 -> 456 | Helped |
| use SimDA instead of Make-A-Video | text-to-video generation · MSR-VTT | CLIPSIM: 0.3049 -> 0.2945 | Hurt |
| use SimDA instead of VideoFactory | text-to-video generation · WebVid validation | FVD: 292.35 -> 363.98 | Hurt |
| use SimDA instead of VideoFactory | text-to-video generation · WebVid validation | CLIPSIM: 0.3070 -> 0.3054 | Hurt |
| use SimDA instead of Tune-A-Video | video editing · Video editing | Frame consistency CLIP Score: 94.1 -> 94.9 | Helped |
| use SimDA instead of Tune-A-Video | video editing · Video editing | Textual alignment CLIP Score: 31.8 -> 31.9 | Helped |
Surprising findings
- Adding adapters to the FFN as well as attention and using only a temporal depth-wise 3D conv adapter gives competitive quality without any new temporal attention layers.
- Video editing from a single text-video pair with SimDA needs 200 tuning steps and about 2.5 minutes versus 500 steps for Tune-A-Video, with a user vote preference of 68.8% for frame consistency.
Limitations
- Relies on a frozen Stable Diffusion backbone, so quality is bounded by the T2I model's capacity.
- Evaluation on MSR-VTT and WebVid uses FVD and CLIPSIM, and the user study covers only a limited set of prompts and compared methods.
- Super-resolution to 1024x1024 is a second-stage model whose quantitative evaluation is not reported in the main text.
Experimental setup
Datasets: WebVid-10M MSR-VTT
Benchmarks: MSR-VTT WebVid validation set
Metrics: FVD CLIPSIM CLIP score user preference parameter count inference speed
Baselines: CogVideo Make-A-Video Imagen Video Video LDM Latent-Shift LVDM VideoFactory ModelScope VDM Tune-A-Video
Builds on: Stable Diffusion Latent-Shift Tune-A-Video Temporal Shift Module Video LDM
Closest papers by meaning
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.89)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.89)
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- VideoDirector: Precise Video Editing via Text-to-Video Models (CVPR 2025 · similarity 0.88)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.88)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.