Research map / Vision-language and multimodal / Multimodal LLMs
DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces customized manga generation and DiffSensei, a SDXL-based diffusion model whose character features are edited by an MLLM (SEED-X) acting as a text-compatible identity adapter, plus masked cross-attention layout control and a dialog embedding, trained on the new MangaZero dataset.
Problem
Story visualization and manga generation lack fine-grained control over multiple characters' appearances, poses, expressions, layouts and dialog placement, and zero-shot customization tends to produce rigid copy-paste results.
Approach
Use an MLLM as a text-compatible character feature adapter that edits source character tokens according to the panel caption before injecting them into a diffusion U-Net via masked cross-attention for per-character layout control, plus a learned dialog-box embedding added to the noised latent.
Key results
- On the MangaZero eval set, DiffSensei reaches CLIP 0.235 and DINO-C 0.651 and dialog F1 0.727, versus 0.229 CLIP / 0.641 DINO-C / 0.720 F1 for MS-Diffusion.
- On unseen Manga109 characters, DiffSensei scores DINO-C 0.600 and F1 0.648, outperforming MS-Diffusion (0.600 / 0.601 F1 on F1).
- Removing the MLLM adapter lowers CLIP by about 1.7% (0.235 to 0.230) and reduces DINO-C in the ablation table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffSensei instead of MS-Diffusion | MangaZero evaluation set · MangaZero evaluation set | CLIP: 0.229 -> 0.235 | Helped |
| Use DiffSensei instead of MS-Diffusion | Manga109 evaluation set · Manga109 evaluation set | F1 score: 0.601 -> 0.648 | Helped |
| Add dialog masked encoding (ablation) | DiffSensei image generator with character masked attention injection · MangaZero evaluation set | F1 score: 0.364 -> 0.653 | Helped |
| Add the Magi image encoder (ablation) | DiffSensei image generator with character and dialog layout modules · MangaZero evaluation set | DINO-I: 0.600 -> 0.618 | Helped |
| Add the MLLM character feature adapter (ablation) | DiffSensei image generator with character masked attention, dialog encoding, and Magi encoder · MangaZero evaluation set | CLIP: 0.231 -> 0.235 | Helped |
| Increase target-panel character-image sampling rate from 0.0 to 1.0 (ablation) | DiffSensei training · MangaZero evaluation set | CLIP: 0.233 -> 0.233; 0.235 at 0.5 | Mixed |
| Set MLLM weighting β to 0.6 or 0.8 rather than 0.4 (ablation) | DiffSensei character feature adapter · MangaZero evaluation set | CLIP: 0.235 -> 0.237 | Helped |
| Increase MLLM weighting β from 0.4 to 1.0 (ablation) | DiffSensei character feature adapter · MangaZero evaluation set | DINO-C: 0.651 -> 0.618 | Hurt |
Surprising findings
- Sampling the source character from the target panel at a high rate hurts text compatibility, so a 0.5 mixing rate was chosen rather than purely random or fully same-panel sampling.
- Completely random source-character sampling made training hard to converge, attributed to manga artistic exaggerations.
Limitations
- Multiple similar input characters can fuse together in generated output, inherited from SDXL.
- Unclear input character images lead to loss of identity.
- Style control is weak when no character image is provided.
Experimental setup
Datasets: MangaZero Manga109
Benchmarks: MangaZero evaluation set Manga109 evaluation set
Metrics: FID CLIP score DINO-I DINO-C F1 score human preference
Baselines: StoryDiffusion AR-LDM StoryGen SEED-Story MS-Diffusion
Builds on: SDXL IP-Adapter-Plus SEED-X MS-Diffusion Magi
Closest papers by meaning
- StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization (AAAI 2025 · similarity 0.87)
- The Manga Whisperer: Automatically Generating Transcriptions for Comics (CVPR 2024 · similarity 0.87)
- StoryGPT-V: Large Language Models as Consistent Story Visualizers (CVPR 2025 · similarity 0.87)
- Storynizor: Consistent Story Generation via Inter-Frame Synchronized and Shuffled ID Injection (AAAI 2025 · similarity 0.86)
- ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026 · similarity 0.86)
- Storybooth: Training-Free Multi-Subject Consistency for Improved Visual Storytelling (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.