Research map / Vision-language and multimodal / Human and embodied modeling
VinaBench: Benchmark for Faithful and Consistent Visual Narratives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VinaBench, a benchmark of ~25K visual-textual narrative pairs annotated with commonsense links and discourse constraints, plus VQA-based alignment/consistency metrics and a ranking metric for evaluating visual narrative generation.
Problem
Visual narrative generation often produces images that are unfaithful to the input text and inconsistent across images (characters, location, time, style), and existing metrics compare against single references and miss consistency.
Approach
Augment visual storytelling datasets with commonsense entity links (from dense captions to narrative phrases) and global/scene-level discourse features (character profiles, appearance style, per-scene characters, time, location), then use these constraints both as training intermediate plans (LLM generates constraints before image generation) and to drive reference-free VQA-based alignment and con
Key results
- Training with LLM-generated constraints (w/ LLM Cons.) improves ARLDM FID from 42.6 to 37.6 on VWP and raises character consistency from 0.376 to 0.689 (VQA-based, MiniCPM-V-2.6).
- Human evaluation on VWP shows MM-Interleaved alignment rising from 2.43 to 3.32 and content consistency from 3.68 to 4.03 (5-point Likert) with LLM constraints, though gold references reach 4.80 overall quality.
- Expert study on 100 samples reports 84.5%-95.5% acceptance rates across constraint types.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LLM-generated narrative constraints | ARLDM · VWP test | CLIP-T-MRR: 0.110 -> 0.151 | Helped |
| Add LLM-generated narrative constraints | ARLDM · VWP test | Character consistency: 0.379 -> 0.551 | Helped |
| Add LLM-generated narrative constraints | StoryGen · VWP test | CLIP-T-MRR: 0.100 -> 0.106 | Helped |
| Add LLM-generated narrative constraints | StoryGen · VWP test | Character consistency: 0.231 -> 0.418 | Helped |
| Add LLM-generated narrative constraints | MM-Interleaved · VWP test | CLIP-T-MRR: 0.066 -> 0.111 | Helped |
| Add LLM-generated narrative constraints | MM-Interleaved · VWP test | Character consistency: 0.582 -> 0.678 | Helped |
| Add gold narrative constraints | MM-Interleaved · VWP test | CLIP-T-MRR: 0.066 -> 0.118 | Helped |
| Remove commonsense links (ablation) | MM-Interleaved with LLM constraints · VWP test | CLIP-T-MRR: 0.111 -> 0.109 | Hurt |
| Remove discourse features (ablation) | MM-Interleaved with LLM constraints · VWP test | CLIP-T-MRR: 0.111 -> 0.107 | Hurt |
| Remove global discourse features (ablation) | MM-Interleaved with LLM constraints · VWP test | CLIP-T-MRR: 0.111 -> 0.110 | Hurt |
| Remove scene-specific discourse features (ablation) | MM-Interleaved with LLM constraints · VWP test | CLIP-T-MRR: 0.111 -> 0.109 | Hurt |
| Shuffle constraints among samples of the same length (ablation) | MM-Interleaved · VWP test | CLIP-T-MRR: 0.066 -> 0.048 | Hurt |
Surprising findings
- Random constraint shuffling (length-matched) performs worse than no constraints, indicating gains come from aligned constraints rather than extra tokens.
Limitations
- Generated visual narratives still fall well short of gold references on all metrics, especially character consistency.
- Constraint labels are produced by LLMs/VLMs; expert check covers only 100 samples (acceptance 84.5-95.5%).
Experimental setup
Datasets: VWP Storyboard20K StorySalon
Benchmarks: VinaBench (VWP) VinaBench (Storyboard20K zero-shot) VinaBench (StorySalon)
Metrics: FID CLIP-I CLIP-T CLIP-T-MRR VQA-MRR VQA-based alignment (non-character entities, character numbe VQA-based consistency (style, character, location) human Likert ratings
Baselines: ARLDM StoryGen MM-Interleaved
Builds on: VQAScore StoryGen ARLDM VWP Storyboard20K
Closest papers by meaning
- ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026 · similarity 0.89)
- SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency (NeurIPS 2025 · similarity 0.88)
- LogiStory: A Logic-Aware Framework for Multi-Image Story Visualization (ICLR 2026 · similarity 0.88)
- Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models (CVPR 2024 · similarity 0.88)
- Consistent Story Generation: Unlocking the Potential of Zigzag Sampling (NeurIPS 2025 · similarity 0.88)
- Storybooth: Training-Free Multi-Subject Consistency for Improved Visual Storytelling (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.