Research map / Video / Video generation
StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StoryDiffusion, a training-free Consistent Self-Attention for consistent multi-image story generation on Stable Diffusion, plus a Semantic Motion Predictor for smooth transition video generation.
Problem
Maintaining subject consistency (identity and attire) across a series of generated images for visual storytelling, and producing smooth long-range transition videos between images.
Approach
Replace self-attention with a version whose keys and values include randomly sampled tokens from other images in the batch, enabling cross-image interaction without training; and predict intermediate transition embeddings in CLIP image-semantic space via a transformer, decoded by an AnimateDiff-based video model through cross-attention.
Key results
- Consistent Self-Attention achieves text-image similarity 0.6586 and character similarity 0.8950, beating IP-Adapter (0.6129, 0.8802) and PhotoMaker (0.6541, 0.8924).
- On transition video generation, the method reaches LPIPS-first 0.3794 and CLIPSIM-frames 0.9870, outperforming SEINE and SparseCtrl on all four metrics.
- In a 30-person user study, StoryDiffusion was preferred 72.8% over IP-Adapter/PhotoMaker for images and 82% over SEINE/SparseCtrl for videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Consistent Self-Attention instead of the original self-attention | StoryDiffusion · Consistent image generation test set | Text-Image Similarity: 0.6541 -> 0.6586 | Helped |
| Use Consistent Self-Attention instead of the original self-attention | StoryDiffusion · Consistent image generation test set | Character Similarity: 0.8924 -> 0.8950 | Helped |
| Use Semantic Motion Predictor for semantic-space transition prediction | StoryDiffusion · Transition video generation test set | LPIPS-frames: 0.1768 -> 0.1635 | Helped |
| Use Semantic Motion Predictor for semantic-space transition prediction | StoryDiffusion · Transition video generation test set | CLIPSIM-frames: 0.9756 -> 0.9870 | Helped |
| Use Consistent Self-Attention with a sampling rate of 0.3 rather than a higher sampling rate (ablation) | StoryDiffusion · Consistent image generation | Subject consistency | Hurt |
| Add PhotoMaker ID-image control to Consistent Self-Attention (ablation) | StoryDiffusion · User-specified ID image generation | ID conformity and image consistency | Helped |
Surprising findings
- A sampling rate of 0.3 failed to maintain subject consistency, while higher rates (default 0.5) preserved it.
Limitations
- Minor clothing details (e.g., ties) may remain inconsistent and need more detailed prompts.
- Not designed for very long videos; sliding window lacks global information exchange.
- Quantitative evaluation of image consistency relies on CLIP similarity on a GPT-4-generated prompt set.
Experimental setup
Datasets: WebVid10M
Benchmarks: internal GPT-4-generated character and activity prompt set transition video test set of ~1000 sampled videos
Metrics: CLIP text-image similarity character similarity (CLIP) LPIPS-first LPIPS-frames CLIPSIM-first CLIPSIM-frames user preference
Baselines: IP-Adapter PhotoMaker SEINE SparseCtrl
Builds on: Stable Diffusion XL AnimateDiff CLIP ControlNet
Closest papers by meaning
- Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models (AAAI 2025 · similarity 0.90)
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR 2024 · similarity 0.90)
- LongDiff: Training-Free Long Video Generation in One Go (CVPR 2025 · similarity 0.89)
- Storybooth: Training-Free Multi-Subject Consistency for Improved Visual Storytelling (ICLR 2025 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.