Research map / Vision-language and multimodal / Visual question answering
Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces XT-VQA, a cross-lingual text-rich VQA benchmark (five extended datasets plus new XPaperQA), analyzes LVLM gaps via mutual information, and proposes MVCL-MI, a KL-distillation fine-tuning of MiniCPM-V to narrow the gap.
Problem
LVLMs perform much worse on text-rich images when the question language differs from the language of text in the image, and existing benchmarks focus on monolingual settings.
Approach
Frames the cross-lingual gap as insufficient activation of visual information (low mutual information between outputs and image), and mitigates it by distilling monolingual output logits into cross-lingual queries via KL minimization combined with cross-lingual cross-entropy.
Key results
- Across 8 open-source LVLMs, average performance drops about 32.5% (Chinese) and 32.6% (French) relative to English-question accuracy on extended datasets.
- MVCL-MI on MiniCPM-V improves cross-lingual XPaperQA-en overall from 22.5 to 25.9 gap-narrowing and XPaperQA-zh cross-lingual accuracy from 29.3 to 21.3 gap reduction (gap in zh paper narrowed from 29.3 to 21.3 points... reported as 10.3 vs 13.5 gap)).
- Removing KL loss drops Chinese-paper overall accuracy from 46.1 to 43.3, showing KL distillation is needed to preserve monolingual performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MVCL-MI to MiniCPM-V | MiniCPM-V · XPaperQA-en (English paper), Chinese questions, overall | F1: 20.4 -> 22.5 | Helped |
| Apply MVCL-MI to MiniCPM-V | MiniCPM-V · XPaperQA-zh (Chinese paper), English questions, overall | F1: 32.6 -> 37.9 | Helped |
| Apply MVCL-MI to MiniCPM-V | MiniCPM-V · XPaperQA-en (English paper), English questions, overall | F1: 25.3 -> 25.9 | Helped |
| Apply MVCL-MI to MiniCPM-V | MiniCPM-V · XPaperQA-zh (Chinese paper), Chinese questions, overall | F1: 46.1 -> 48.2 | Helped |
| Add cross-lingual cross-entropy loss (ablation) | MVCL-MI on MiniCPM-V · XPaperQA-en (English paper), Chinese questions, overall | F1: 21.7 -> 22.5 | Helped |
| Add cross-lingual cross-entropy loss (ablation) | MVCL-MI on MiniCPM-V · XPaperQA-zh (Chinese paper), English questions, overall | F1: 33.5 -> 37.9 | Helped |
| Add KL loss (ablation) | MVCL-MI on MiniCPM-V · XPaperQA-en (English paper), Chinese questions, overall | F1: 19.7 -> 22.5 | Helped |
| Add KL loss (ablation) | MVCL-MI on MiniCPM-V · XPaperQA-zh (Chinese paper), Chinese questions, overall | F1: 43.3 -> 48.2 | Helped |
| Add KL loss (ablation) | MVCL-MI on MiniCPM-V · XPaperQA-zh (Chinese paper), English questions, overall | F1: 31.9 -> 37.9 | Helped |
| Add KL loss (ablation) | MVCL-MI on MiniCPM-V · XPaperQA-en (English paper), English questions, overall | F1: 25.3 -> 25.3 | No clear effect |
| Use cross-lingual instructions instead of source-language instructions | LVLMs on XT-VQA · XT-VQA, Chinese questions versus English/source-language questions | accuracy: 32.5% | Hurt |
| Use cross-lingual instructions instead of source-language instructions | LVLMs on XT-VQA · XT-VQA, French questions versus English/source-language questions | accuracy: 32.6% | Hurt |
Surprising findings
- Models with multilingual instruction tuning (Cog-VLM, MiniCPM-V) show more consistent cross-lingual performance than LLaVA-v1.6-34b.
- Removing KL loss makes cross-lingual performance worse than the original model on Chinese papers.
Limitations
- Closed-source models evaluated only on a subset.
- XPaperQA construction relies on Gemini-Pro generation with 87% manually verified accuracy.
- Mutual information analysis is approximated using Gaussian-noise images and limited to 100 ChartQA examples.
Experimental setup
Datasets: OCRVQA TextVQA ChartQA DocVQA QASPER XPaperQA
Benchmarks: XT-VQA OCRVQA TextVQA ChartQA DocVQA XPaperQA
Metrics: accuracy F1 score
Baselines: LLaVA-v1.5-13b LLaVA-v1.6-34b InstructBLIP mPLUG-Owl2 Qwen-VL-Chat Monkey CogVLM MiniCPM-V GPT-4o Gemini-1.5-flash
Builds on: visual contrastive decoding QASPER
Closest papers by meaning
- Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.91)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.90)
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · similarity 0.90)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.90)
- Learning Interleaved Image-Text Comprehension in Vision-Language Large Models (ICLR 2025 · similarity 0.90)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.