Research map / Large language models / Language, safety and interpretability
StoryGPT-V: Large Language Models as Consistent Story Visualizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StoryGPT-V, a two-stage model pairing a character-aware latent diffusion model (with segmentation-supervised cross-attention) and an LLM (OPT-6.7B) aligned to its input space to resolve co-referential pronouns for consistent story visualization.
Problem
Story visualization requires resolving pronouns (he, she, they) across frame descriptions and keeping characters and backgrounds consistent across frames, which existing text-to-image and memory-based autoregressive diffusion models handle poorly, with high memory cost for long stories.
Approach
Augment CLIP text token embeddings with character visual features and supervise cross-attention with SAM character masks in a first-stage LDM; then train an LLM with interleaved image-text input to output [IMG] tokens that are mapped to the first-stage fused embedding space, letting the LLM implicitly resolve references while keeping memory low by storing frames as a few tokens.
Key results
- On FlintstonesSV with co-referential text, StoryGPT-V reaches 88.45 Char-Acc and 21.71 FID, versus 82.43 Char-Acc and 36.29 FID for Story-LDM.
- On PororoSV with co-referential text, Char-Acc improves to 36.06 from 29.14 for Story-LDM.
- StoryGPT-V sustains 80.92 Char-Acc at 50 frames where Story-LDM runs out of 80GB GPU memory beyond 40 frames.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add segmentation-mask supervision for cross-attention (Lreg) (ablation) | Char-LDM · FlintStonesSV | Char-Acc: 88.86 -> 90.36 | Helped |
| add character visual features to text embeddings (ablation) | Char-LDM · FlintStonesSV | Char-Acc: 87.45 -> 90.36 | Helped |
| allow visual encoder fine-tuning instead of freezing it (ablation) | Char-LDM · FlintStonesSV | Char-Acc: 88.67 -> 90.36 | Helped |
| use interleaved image-text input instead of caption-only input (ablation) | StoryGPT-V · FlintStonesSV with co-reference | Char-Acc: 71.77 -> 87.96 | Helped |
| align the output to fused character-augmented embeddings instead of text embeddings (ablation) | StoryGPT-V with interleaved input · FlintStonesSV with co-reference | Char-Acc: 86.10 -> 87.96 | Helped |
| add pixel-level Limg loss (ablation) | StoryGPT-V with interleaved input and fused-embedding alignment · FlintStonesSV with co-reference | Char-Acc: 87.96 -> 88.45 | Helped |
| use StoryGPT-V instead of Story-LDM | FlintStonesSV with co-reference · FlintStonesSV with co-reference | Char-Acc: 77.23 -> 88.45 | Helped |
| use StoryGPT-V instead of Story-LDM | PororoSV with co-reference · PororoSV with co-reference | FID: 26.64 -> 19.56 | Helped |
| use StoryGPT-V instead of Story-LDM for longer story generation | FlintStonesSV · 40-frame stories with referential text | Char-Acc: 63.40 -> 81.04 | Helped |
Limitations
- Evaluation relies on a BLIP-2 captioner fine-tuned on the datasets as a text-image alignment judge rather than standard CLIP score
- Story-LDM is pretrained by the authors because no checkpoint was released, and several baselines' code was not public so results were sourced from papers
Experimental setup
Datasets: FlintstonesSV PororoSV
Benchmarks: FlintstonesSV PororoSV
Metrics: Char-Acc Char-F1 BG-Acc BG-F1 FID BLEU4 CIDEr
Baselines: StoryDALL-E LDM Story-LDM AR-LDM StoryImager ContextualStory
Builds on: Stable Diffusion Story-LDM BLIP-2 FROMAGe
Closest papers by meaning
- ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026 · similarity 0.91)
- Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models (CVPR 2024 · similarity 0.90)
- Boosting Consistency in Story Visualization with Rich-Contextual Conditional Diffusion Models (AAAI 2025 · similarity 0.89)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.89)
- Storybooth: Training-Free Multi-Subject Consistency for Improved Visual Storytelling (ICLR 2025 · similarity 0.89)
- StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.