Research map / Vision-language and multimodal / Multimodal LLMs

CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CVLUE, a Chinese vision-language benchmark with culturally representative images collected by Chinese native speakers across four tasks (ITR, VQA, VG, VD), and benchmarks multilingual VLMs to show a large English-Chinese gap and cultural knowledge deficits.

Problem

Existing Chinese vision-language datasets reuse Western-centric images from English datasets, so they cannot evaluate VLMs on Chinese cultural understanding, and Chinese VL benchmarks cover only single task types.

Approach

Select 92 culturally representative object categories with Chinese native speakers and collect new images from the Chinese Internet, then annotate four VL tasks (image-text retrieval, VQA, visual grounding, visual dialogue) with multi-stage quality control, and analyze VLM performance by cultural category.

Key results

• Zero-shot and fine-tuned VLMs score far lower on CVLUE than on English counterparts; e.g., QwenVL VQA accuracy 29.9 on CVLUE vs 78.0 on VQA-v2, and mPLUG-Owl2 gets 20.4 on CVLUE VQA vs 79.2 on VQA-v2.
• On the CVLUE VG task, QwenVL averages 34.8 IoU on categories closely tied to Chinese culture versus 46.9 on MS-COCO-overlapping categories.
• Fine-tuning X2VLM on CVLUE raises ITR TR R@1 to 54.8 and VG IoU to 48.8, and fine-tuned models outperform zero-shot ones on Chinese-culture categories.

Measured effects reported in the paper

Surprising findings

• Zero-shot large VLMs with more parameters often underperform smaller fine-tuned models on CVLUE, while they sometimes win on English tasks.
• Translating the VG test set to English does not help culturally specific categories, suggesting the English side of VLMs also lacks this knowledge.

Limitations

• Only four tasks and a small set of open-source VLMs were evaluated due to budget constraints.
• Default hyperparameters were used for all baselines without tuning, so results may not reflect full model potential.
• Baseline comparisons are limited to models that could be run; larger VLMs were not fine-tuned.

Experimental setup

Datasets: CVLUE COCO (5K) VQA-v2 RefCOCOg Visdial 1.0

Benchmarks: CVLUE COCO (5K) VQA-v2 RefCOCOg Visdial 1.0

Metrics: R@1 R@5 R@10 accuracy IoU

Baselines: CCLM X2VLM Qwen-VL Qwen-VL-Chat mPLUG-Owl2

Builds on: MaRVL MS-COCO Visdial 1.0 RefCOCOg

Closest papers by meaning

• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.90)
• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.89)
• See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.89)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.87)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.87)