Research map / Image generation / Fast diffusion sampling
Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RCDMs, a two-stage story visualization method that first predicts CLIP frame embeddings of unknown clips with a frame-prior transformer diffusion model, then generates all frames in one pass with a frame-contextual 3D diffusion model (Stable Diffusion 1.5 based) conditioned on known-clip images and text/frame embeddings.
Problem
Story visualization requires consistent characters, style, and narrative across multiple frames, but autoregressive caption-dependent methods suffer from weak contextual conditioning and inconsistency.
Approach
Predict semantic (CLIP) embeddings of the unknown clip from captions and known frames, then jointly inject known-clip reference images (image level) and predicted frame embeddings plus all-caption text embeddings (feature level) into a 3D frame-attention diffusion model that generates the whole story in a single forward pass.
Key results
- On FlintstonesSV, RCDMs achieve FID 14.96, Char-Acc 78.44, Char-F1 85.51, improving over AR-LDM (19.28 / 62.58 / 79.25).
- On PororoSV, RCDMs reach FID 16.25 versus AR-LDM 17.40, with Char-Acc 41.48 and Char-F1 59.03.
- Inference takes 12.7 s per story versus 14.6 s for AR-LDM and 16.9 s for Story-LDM on a V100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RCDMs instead of AR-LDM | Story visualization · FlintstonesSV | FID: 19.28 -> 14.96 | Helped |
| use RCDMs instead of AR-LDM | Story visualization · PororoSV | Char-Acc: 35.18 -> 41.48 | Helped |
| add image-level condition (ablation) | B0 on FlintstonesSV · FlintstonesSV | Char-Acc: 56.44 -> 61.88 | Helped |
| add MIM to B1 (ablation) | RCDMs on FlintstonesSV · FlintstonesSV | Char-Acc: 61.88 -> 72.53 | Helped |
| add SSM to B1 (ablation) | RCDMs on FlintstonesSV · FlintstonesSV | Char-Acc: 61.88 -> 73.58 | Helped |
| remove the stage-1 frame-prior model (ablation) | RCDMs on FlintstonesSV · FlintstonesSV | Char-Acc: 78.44 -> 75.73 | Hurt |
| use RCDMs instead of AR-LDM | Story inference · Story inference | inference time: 14.6 seconds -> 12.7 seconds | Helped |
Limitations
- Works only on closed-set datasets with a limited set of characters and scenes.
- Evaluation limited to two cartoon datasets (FlintstonesSV, PororoSV).
Experimental setup
Datasets: FlintstonesSV PororoSV
Benchmarks: FlintstonesSV PororoSV
Metrics: FID Char-Acc Char-F1 user preference
Baselines: LDM StoryGAN Story-DALL-E Story-LDM AR-LDM CP-CSV DuCo-StoryGAN
Builds on: unCLIP Stable Diffusion AR-LDM classifier-free guidance
Closest papers by meaning
- StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation (NeurIPS 2024 · similarity 0.90)
- StoryGPT-V: Large Language Models as Consistent Story Visualizers (CVPR 2025 · similarity 0.89)
- ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context (AAAI 2025 · similarity 0.89)
- Storybooth: Training-Free Multi-Subject Consistency for Improved Visual Storytelling (ICLR 2025 · similarity 0.88)
- SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency (NeurIPS 2025 · similarity 0.88)
- One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.