Research map / Vision-language and multimodal / Multimodal LLMs

Culture In a Frame: C$^3$B as a Comic-Based Benchmark for Multimodal Culturally Awareness

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces C$^3$B, a comic-based multicultural, multitask, multilingual benchmark (2220 images, ~18.8k QA pairs) with three escalating tasks for evaluating MLLM cultural awareness, and benchmarks 11 open-source MLLMs against human performance.

Problem

Existing cultural awareness benchmarks for MLLMs use real-world images (usually one culture per image), mostly have single-question samples, lack progressive difficulty, and are weak on cross-lingual evaluation.

Approach

Use comics (largely AI-generated fictional scenes, plus Manga109 pages) to pack multiple cultures into one image, and structure the benchmark as a chain of three tasks of rising difficulty: culture-aware object extraction, cultural-conflict detection/description, and multilingual culturally-aligned translation.

Key results

• Qwen2.5-VL achieves the best Extraction@Culture Q1 accuracy (53.7) and best CACC on Conflict@Culture Q4 (34.2) among 11 MLLMs.
• Under the human-model analysis, human Q4 ACC is 60.0/45.0/35.0 on easy/medium/hard tiers versus at most 1.90 for models.
• C$^3$B reports CDPI 2.28 and CAD 14.29, higher than CVQA (4.91), CulturalVQA (3.58), MOSAIC-1.5k (5.46) and GIMMICK (7.18).

Surprising findings

• Adding ground-truth-like Q1 and Q2 answers to the Q4 prompt yields only marginal gains for models (+0.003 for Q1 alone, none for Q2 alone), despite moderate correlations (0.56, 0.51).
• LLaVA-NeXT tends to describe the image rather than answer questions directly, and LLaVA1.5-7B outputs 'A' for 78.4% of Q2 answers.

Limitations

• Comic images for Extraction and Conflict tasks are AI-generated (Doubao) rather than real comics, and the Generation task uses only Japanese source text from Manga109.
• Evaluation only covers 11 open-source MLLMs; no proprietary models were tested.
• Human evaluation involved a small number of volunteers (three per study, 100 questions per tier).

Experimental setup

Datasets: Manga109

Benchmarks: C$^3$B

Metrics: ACC CACC BLEU COMET BLEURT CDPI CBI CAD inter-rater agreement

Baselines: SPHINX Monkey MiniGPT-v2 mPLUG-Owl3 LLaVA1.5-7B LLaVA-NeXT LLaVA-OV InternLM-XC2.5 Llama3.2 Qwen2.5-VL InternVL2

Builds on: Manga109 CVQA GIMMICK

Closest papers by meaning

• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.90)
• Beyond Aesthetics: Cultural Competence in Text-to-Image Models (NeurIPS 2024 · similarity 0.90)
• All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages (CVPR 2025 · similarity 0.90)
• No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.88)
• World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models (CVPR 2026 · similarity 0.88)