Research map / Vision-language and multimodal / Human and embodied modeling
The Manga Whisperer: Automatically Generating Transcriptions for Comics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Magi, a unified DETR-style transformer that detects manga panels, text blocks and characters, clusters characters and links texts to speakers, then sorts text into reading order to produce page-level dialogue transcripts.
Problem
Making manga accessible to people with visual impairments requires automatically generating a transcript of who said what and when (diarisation) from manga pages, which needs detection, character clustering, and speaker association.
Approach
Treats detection and association as graph generation over the whole page: [OBJ] tokens are nodes and special [C2C]/[T2C] tokens predict character-character and text-character edges, with a topological-sort panel ordering for reading order.
Key results
- Magi reaches 0.6551 AMI on PopManga Test-S for character clustering versus 0.1550 for AMFR, the best prior baseline.
- Magi achieves 0.9227 text detection AP on PopManga Test-S versus 0.7922 for Grounding-DINO.
- Speaker association Recall@#text of 0.8448 on PopManga Test-S versus 0.7758 for a shortest-distance heuristic.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Magi instead of DASS for character detection | Magi · PopManga (Test-S) | average precision (character): 0.8410 -> 0.8485 | Helped |
| use Magi instead of Grounding-DINO for text detection | Magi · PopManga (Test-S) | average precision (text): 0.7922 -> 0.9227 | Helped |
| use Magi instead of DASS for character detection | Magi · PopManga (Test-U) | average precision (character): 0.8580 -> 0.8615 | Helped |
| use Magi instead of Grounding-DINO for text detection | Magi · PopManga (Test-U) | average precision (text): 0.8301 -> 0.9208 | Helped |
| use Magi instead of DASS for body-character detection | Magi · Manga109 | average precision (body character): 0.9251 -> 0.9015 | Hurt |
| use Magi instead of Grounding-DINO for panel detection | Magi · Manga109 | average precision (panel): 0.5131 -> 0.9357 | Helped |
| use full-page context rather than crop-only embeddings for character clustering (ablation) | Magi · PopManga (Test-S) | AMI: 0.4342 -> 0.6551 | Helped |
| use full-page context rather than crop-only embeddings for character clustering (ablation) | Magi · PopManga (Test-U) | AMI: 0.4387 -> 0.6509 | Helped |
| use full-page context rather than crop-only embeddings for character clustering (ablation) | Magi · Manga109 (face) | AMI: 0.3182 -> 0.5369 | Helped |
| use full-page context rather than crop-only embeddings for character clustering (ablation) | Magi · Manga109 (body) | AMI: 0.5132 -> 0.6345 | Helped |
| use Magi instead of the shortest-distance heuristic for speaker association | Magi · PopManga (Test-S) | Recall@#text: 0.7758 -> 0.8448 | Helped |
| use Magi instead of the shortest-distance heuristic for speaker association | Magi · PopManga (Test-U) | Recall@#text: 0.7659 -> 0.8313 | Helped |
Surprising findings
- Operating on crops alone (Magi crop-only) is clearly worse than the full-page model, supporting the value of page context for clustering.
Limitations
- Character matching can confuse visually similar characters (e.g., brothers in matching clothing).
- Text-to-speaker association uses low-confidence thresholds and non-dialogue texts remain ambiguous.
- Pseudo-annotations for Mangadex-1.5M are noisy and text-character labels are heuristic.
Experimental setup
Datasets: PopManga Mangadex-1.5M Manga109
Benchmarks: PopManga (Test-S) PopManga (Test-U) Manga109
Metrics: average precision AMI NMI MRR MAP@R P@1 R-Precision Recall@#text
Baselines: DASS Grounding-DINO AMFR ZACI CLIP DINO-V2 shortest distance heuristic
Builds on: Conditional-DETR DETR MAE TrOCR
Closest papers by meaning
- DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation (CVPR 2025 · similarity 0.87)
- Advancing Manga Analysis: Comprehensive Segmentation Annotations for the Manga109 Dataset (CVPR 2025 · similarity 0.84)
- Parameterized Synthetic Text Generation with SimpleStories (NeurIPS 2025 · similarity 0.83)
- One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt (ICLR 2025 · similarity 0.83)
- Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models (CVPR 2024 · similarity 0.83)
- StoryGPT-V: Large Language Models as Consistent Story Visualizers (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.