Research map / Video / Video generation
ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ContextualStory, a non-autoregressive latent diffusion model for story visualization and continuation that adds spatially-enhanced temporal attention, a transformer-based storyline contextualizer, and a storyflow adapter on top of Stable Diffusion 2.1-base.
Problem
Visual storytelling requires generating coherent frame sequences from a multi-sentence storyline while keeping characters and scenes consistent; existing autoregressive diffusion methods are memory-intensive, slow, and have limited context integration.
Approach
Replaces autoregressive frame-sentence conditioning with a non-autoregressive UNet that uses temporal convolutions and local-window temporal attention (SETA) across frames, a storyline contextualizer that propagates context across CLIP sentence embeddings, and a storyflow adapter that conditions on L2 differences between adjacent frames.
Key results
- On PororoSV story visualization, ContextualStory reaches FID 13.61, Char. F1 77.24, and Frm. Acc. 51.59, beating AR-LDM (FID 16.59) and StoryImager (FID 15.63).
- On FlintstonesSV story continuation, FID 13.27, Char. F1 91.29, Frm. Acc. 81.91, outperforming RCDMs (FID 14.96).
- Inference uses 5 GB memory and 11.8 s versus 40 GB and 40.4 s for AR-LDM on an A800 with 50 DDIM steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add StoryFlow Adapter (ablation) | ContextualStory · PororoSV story visualization | FID: 14.84 -> 13.61 | Helped |
| add Storyline Contextualizer (ablation) | ContextualStory · PororoSV story visualization | FID: 15.02 -> 13.61 | Helped |
| add SETA (ablation) | ContextualStory · PororoSV story visualization | FID: 17.42 -> 13.61 | Helped |
| add Temporal Convolution (ablation) | ContextualStory · PororoSV story visualization | FID: 19.69 -> 13.61 | Helped |
| replace SETA with Vanilla Temporal Attention (ablation) | ContextualStory · PororoSV story visualization | FID: 13.61 -> 14.78 | Hurt |
| use the proposed temporal-attention Storyline Contextualizer instead of Simplified, Dual Self-Attention, or Causal variants (ablation) | ContextualStory · PororoSV story visualization | FID: 13.94; 13.75; 14.59 -> 13.61 | Helped |
| increase Storyline Contextualizer layers from 1 to 4 (ablation) | ContextualStory · PororoSV story visualization | FID: 13.96 -> 13.61 | Helped |
| increase Storyline Contextualizer layers from 4 to 8 (ablation) | ContextualStory · PororoSV story visualization | FID: 13.61 -> 14.78 | Hurt |
| use zero instead of random initialization for Storyline Contextualizer (ablation) | ContextualStory · PororoSV story visualization | FID: 13.84 -> 13.61 | Helped |
| use zero instead of random initialization for StoryFlow Adapter (ablation) | ContextualStory · PororoSV story visualization | FID: 14.66 -> 13.61 | Helped |
| process the first image with convolution in each UNet block instead of directly concatenating its latent (ablation) | ContextualStory · PororoSV story continuation | FID: 14.17 -> 13.86 | Helped |
| omit reference text (ablation) | ContextualStory · PororoSV story visualization | Char. F1: 69.65 -> 77.24 | Helped |
Surprising findings
- StoryGPT-V scores higher than ContextualStory on FlintstonesSV with reference text (Char. F1 94.17 vs 88.67), which the authors attribute to LLM-based reference resolution.
Limitations
- Struggles to maintain character layout and detail when frames contain many characters, inherited from Stable Diffusion 2.1-base.
- Reference-text setting shows a drop in Char. F1 and Frm. Acc. relative to the name-based setting.
Experimental setup
Datasets: PororoSV FlintstonesSV
Benchmarks: PororoSV FlintstonesSV
Metrics: FID Character F1 Frame accuracy
Baselines: StoryGAN StoryDALL-E MEGA-StoryDALL-E Story-LDM AR-LDM StoryImager RCDMs StoryGPT-V
Builds on: Stable Diffusion 2.1 AR-LDM Story-LDM PYoCo
Closest papers by meaning
- StoryGPT-V: Large Language Models as Consistent Story Visualizers (CVPR 2025 · similarity 0.89)
- Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models (AAAI 2025 · similarity 0.89)
- SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency (NeurIPS 2025 · similarity 0.88)
- ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026 · similarity 0.88)
- Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models (CVPR 2024 · similarity 0.87)
- LogiStory: A Logic-Aware Framework for Multi-Image Story Visualization (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.