Research map / Vision-language and multimodal / Human and embodied modeling
StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StoryWeaver, a Stable Diffusion-based story character customization model that uses a Character-Graph to build detailed captions and knowledge-enhanced spatial guidance to reduce identity blending in multi-character scenes, evaluated on a new TBC-Bench.
Problem
Story visualization struggles to balance character identity preservation with text-semantic alignment, and multi-character generation suffers from identity blending in cross-attention.
Approach
A Character-Graph (objects, attributes, events) built from VLM captions and a scene-graph parser provides detailed structured captions for training and inference, and a knowledge-enhanced spatial guidance module refines cross-attention maps with Gaussian position priors modulated by character descriptions.
Key results
- Single-character on Frozen: DINO-I 62.17 and CLIP-T 36.74 for StoryWeaver, outperforming Dreambooth (55.01 / 27.12) and other baselines.
- Multi-character on Pororo: CLIP-T 34.30, F-Acc 40.45, C-F1 59.72, versus LoRA-Composer C-F1 47.36.
- Abstract reports average gains of +9.03% DINO-I and +13.44% CLIP-T over compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Character-Graph customization (C-CG) and knowledge-enhanced spatial guidance (KE-SG) (ablation) | StoryWeaver · Pororo single-character | DINO-I: 39.57 -> 64.96 | Helped |
| Add Character-Graph customization (C-CG) and knowledge-enhanced spatial guidance (KE-SG) (ablation) | StoryWeaver · Pororo single-character | CLIP-T: 28.74 -> 33.26 | Helped |
| Add Character-Graph customization (C-CG) and knowledge-enhanced spatial guidance (KE-SG) (ablation) | StoryWeaver · Pororo multi-character | C-F1: 48.26 -> 59.72 | Helped |
| Add KE-SG (ablation) | StoryWeaver · Pororo multi-character | F-Acc: 35.73 -> 40.45 | Helped |
| Add KE-SG (ablation) | StoryWeaver · Pororo multi-character | C-F1: 52.69 -> 59.72 | Helped |
| Add KE-SG (ablation) | StoryWeaver · Pororo single-character | CLIP-T: 32.17 -> 33.26 | Helped |
| Use StoryWeaver instead of the strongest listed baseline | Pororo single-character · Pororo single-character | CLIP-T: 29.98 -> 33.26 | Helped |
| Use StoryWeaver instead of the strongest listed baseline | Frozen single-character · Frozen single-character | DINO-I: 60.87 -> 62.17 | Helped |
| Use StoryWeaver instead of the strongest listed baseline | Pororo multi-character · Pororo multi-character | C-F1: 47.36 -> 59.72 | Helped |
| Use StoryWeaver instead of the strongest listed baseline | Frozen multi-character · Frozen multi-character | CLIP-T: 30.71 -> 34.94 | Helped |
Surprising findings
- IP-Adapter (plus) achieves highest identity similarity on single-character tasks but the lowest CLIP-T, indicating a strong identity-semantics trade-off.
Limitations
- Dataset is small (about 10 images per character) and limited to two cartoon series, collected due to resource constraints.
- Relies on Gaussian position priors and manually set hyperparameters (alpha, beta), with layout assumed evenly distributed horizontally by default.
- Evaluation uses CLIP-T and DINO-based classifiers with a fixed 0.5 threshold and a small set of stories.
Experimental setup
Datasets: Pororo and Frozen frames collected from YouTube PororoSV
Benchmarks: TBC-Bench
Metrics: DINO-I CLIP-I CLIP-T F-Acc C-F1 user study T-A/I-A/V-Q model parameters per dataset
Baselines: StoryGEN IP-Adapter (base) IP-Adapter (plus) LoRA Dreambooth Mix-of-Show LoRA-Composer
Builds on: ERNIE Dreambooth LoRA IP-Adapter Mix-of-Show
Closest papers by meaning
- StoryGPT-V: Large Language Models as Consistent Story Visualizers (CVPR 2025 · similarity 0.89)
- ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026 · similarity 0.87)
- DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation (CVPR 2025 · similarity 0.87)
- LogiStory: A Logic-Aware Framework for Multi-Image Story Visualization (ICLR 2026 · similarity 0.87)
- ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context (AAAI 2025 · similarity 0.85)
- Storynizor: Consistent Story Generation via Inter-Frame Synchronized and Shuffled ID Injection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.