Research map / Vision-language and multimodal / Multimodal LLMs

IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces IndicVisionBench, a culturally grounded benchmark of ~5K Indian images and 37K+ QA pairs across VQA, OCR, and multimodal MT in English and 10 Indic languages, and evaluates 8 VLMs.

Problem

Existing VLM benchmarks are Western-centric, leaving performance on culturally diverse, multilingual Indian settings (Indic scripts, Indian cultural concepts) poorly measured.

Approach

A culturally grounded, India-centric multimodal benchmark spanning three tracks (VQA with six question types including adversarial false-premise questions, OCR on Wikisource pages, and image-grounded MMT) across English and 10 Indic languages, with parallel annotations for cross-lingual analysis.

Key results

• Gemini-2.5 leads all tracks; on VQA-Parallel English long-answer it scores 9.45/10 versus 8.66 for GPT-4o and 8.70 for Gemma-3, and on adversarial questions 5.79 versus 2.95 for GPT-4o.
• On MMT, Gemini-2.5 reaches BLEU 53.27 in Gujarati and 44.51 in Bengali, while 7B open models score far lower (e.g., Maya Hindi BLEU 18.09, Chitrarth-1 Bengali BLEU 21.89).
• On OCR, Gemini-2.5 achieves median-free word-level ANLS-based lead across languages, with Malayalam the hardest (59.64 ANLS-W), and GPT-4o performing unexpectedly poorly despite high expectations.

Surprising findings

• GPT-4o performs poorly on OCR for Indic scripts despite strong general capability, falling below Chitrapathak and LLaMA-4 in several languages.
• Removing the image input causes only modest drops for some models, but the paper reports images are necessary for short-answer questions; Gemini-2.5 scores on adversarial questions remain very low in absolute terms.

Limitations

• Language coverage limited for the lowest-resource Indic languages and topic diversity could be expanded.
• Human annotations and LLM-based scoring may not fully capture cultural grounding; LLM-as-judge used for open-ended QA.
• Visual modality effect on MMT not studied.

Experimental setup

Datasets: Wikisource pages (OCR) Creative Commons web images (VQA) crowd-sourced volunteer images (VQA) Visual Genome (referenced for prior MMT work)

Benchmarks: IndicVisionBench-VQA IndicVisionBench-VQA-Parallel IndicVisionBench-VQA-Indic IndicVisionBench-MMT IndicVisionBench-OCR

Metrics: Exact Match LLM-as-judge score (0-10) BLEU RIBES ANLS WER CER

Baselines: Maya PALO Pangea Chitrarth-1 Chitranuvad LLaMA-4-Maverick Gemma-3-27B GPT-4o Chitrapathak

Builds on: LLaVA OCRBenchv2

Closest papers by meaning

• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.92)
• Culture In a Frame: C$^3$B as a Comic-Based Benchmark for Multimodal Culturally Awareness (ICLR 2026 · similarity 0.90)
• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.90)
• Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation (ICLR 2026 · similarity 0.90)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
• See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.90)