Research map / Image generation / Text-to-image diffusion
Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StoryGen, a stable-diffusion-based auto-regressive model with a vision-language context module that conditions on diffusion features of preceding image-caption pairs, and releases the StorySalon dataset for open-ended visual storytelling.
Problem
Generating coherent image sequences from a given storyline for unseen characters without test-time optimization, a setting where prior story visualization methods overfit to closed sets of characters.
Approach
Inserts a parallel image cross-attention layer into the SD UNet that attends to diffusion denoising features of preceding frames (extracted with a small timestep t/10 and captioned by their text), enabling learning-based generalization to unseen characters without per-character LoRA tuning.
Key results
- StoryGen achieves FID 33.90, CLIP-I 0.7467 on StorySalon test set, versus 42.01 FID for Finetuned-SDM and 38.34 for StoryDALL-E.
- In human evaluation for story continuation, StoryGen gets 96.87% preference versus 2.50% for AR-LDM and 0.63% for StoryDALL-E.
- StoryGen has a slightly lower CLIP-T (0.2875) than Stable Diffusion baselines (0.3218 for SDM), which the authors attribute to CLIP's bias toward natural images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the visual-language context module (ablation) | StoryGen · StorySalon test set | FID: 38.81 -> 33.90 | Helped |
| Use text-guided diffusion features instead of VAE features for context (ablation) | StoryGen · StorySalon test set | FID: 36.98 -> 33.90 | Helped |
| Use text-guided diffusion features instead of CLIP image embeddings for context (ablation) | StoryGen · StorySalon test set | FID: 36.66 -> 33.90 | Helped |
| Use text-guided diffusion features instead of BLIP image-encoder features for context (ablation) | StoryGen · StorySalon test set | FID: 34.78 -> 33.90 | Helped |
| Use a smaller preceding-frame diffusion timestep rather than t′ = t (ablation) | StoryGen · StorySalon test set | FID: 36.41 -> 33.90 | Helped |
| Use a smaller preceding-frame diffusion timestep rather than t′ = t (ablation) | StoryGen · StorySalon test set | CLIP-I: 0.7141 -> 0.7467 | Helped |
| Use preceding image-caption context (ablation) | StoryGen · StorySalon test set | CLIP-I: 0.6869 -> 0.7467 | Helped |
| Generate with StoryGen rather than StoryDALL·E | StorySalon test set · StorySalon test set | FID: 38.34 -> 33.90 | Helped |
| Generate with StoryGen rather than SDM | StorySalon test set · StorySalon test set | CLIP-T: 0.3218 -> 0.2875 | Hurt |
| Use StoryGen rather than SDM | StoryGen · Story generation | Preference: 14.05% -> 67.14% | Helped |
| Use StoryGen rather than AR-LDM | StoryGen · Story continuation | Preference: 2.50% -> 96.87% | Helped |
Surprising findings
- The paper reports a lower CLIP-T score for StoryGen than the plain SDM baseline, attributed to the text-image conflict introduced by the image condition.
Limitations
- CLIP-T text alignment is lower than the SDM baselines, reflecting the trade-off with the image condition.
- Evaluation relies largely on human ratings and CLIP-based metrics; no standardized consistency metric is available.
- Dataset is dominated by animation-style images, which may limit realistic-style generation.
Experimental setup
Datasets: StorySalon MS-COCO PororoSV FlintstonesSV DiDeMoSV VIST
Benchmarks: StorySalon test set
Metrics: FID CLIP-I CLIP-T human rating (alignment, style, content, character, quality) human preference
Baselines: Stable Diffusion Prompt-SDM Finetuned-SDM StoryDALL-E AR-LDM
Builds on: Stable Diffusion AR-LDM StoryDALL-E ControlNet classifier-free guidance
Closest papers by meaning
- StoryGPT-V: Large Language Models as Consistent Story Visualizers (CVPR 2025 · similarity 0.90)
- VinaBench: Benchmark for Faithful and Consistent Visual Narratives (CVPR 2025 · similarity 0.88)
- One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt (ICLR 2025 · similarity 0.88)
- ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026 · similarity 0.88)
- Storybooth: Training-Free Multi-Subject Consistency for Improved Visual Storytelling (ICLR 2025 · similarity 0.88)
- ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.