Research map / Vision-language and multimodal / Multimodal LLMs
II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces II-Bench, a 1,222-image, 1,434-question multiple-choice benchmark for image implication understanding, and evaluates 20 MLLMs and humans, finding a large gap (best model ~74.8% vs human avg 90.3%).
Problem
Existing MLLM benchmarks mostly test knowledge or simple image understanding, leaving higher-order perception of image implications (metaphor, symbolism, emotion) under-evaluated.
Approach
A benchmark of human-curated implication images (comics, memes, posters, art) with six-option questions targeting metaphors and hidden meanings, plus prompt ablations adding emotion, domain, or rhetoric hints to probe model weaknesses.
Key results
- Best MLLM Qwen-VL-MAX reaches 74.8% overall accuracy versus 90.3% human average (98.2% best human) on II-Bench.
- Best open-source model LLaVA-1.6-34B scores 73.8%, only about 1 point below the top closed-source model.
- Adding emotion polarity hints raises LLaVA-1.6-34B from 73.8% to 75.3% and most models improve, while CoT generally does not help (e.g., CogVLM2 70.3% to 69.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image emotion polarity information to the prompt (ablation) | MLLMs on II-Bench · II-Bench test | accuracy: 74.8 -> 75.5 | Helped |
| add image emotion polarity information to the prompt (ablation) | LLaVA-1.6-34B · II-Bench test | accuracy: 73.8 -> 75.3 | Helped |
| add image domain information to the prompt (ablation) | Qwen-VL-MAX · II-Bench test | accuracy: 74.8 -> 74.1 | No clear effect |
| add image rhetoric information to the prompt (ablation) | Qwen-VL-MAX · II-Bench test | accuracy: 74.8 -> 73.6 | Hurt |
| add chain-of-thought prompting (ablation) | CogVLM2-Llama3-Chat-19B · II-Bench test | accuracy: 70.3 -> 69.3 | Hurt |
| add chain-of-thought prompting (ablation) | LLaVA-1.6-34B · II-Bench test | accuracy: 73.8 -> 60.0 | Hurt |
| add chain-of-thought prompting (ablation) | GPT-4o · II-Bench test | accuracy: 72.6 -> 75.7 | Helped |
| add one in-context example (ablation) | Qwen-VL-Chat · II-Bench test | accuracy: 53.4 -> 43.3 | Hurt |
| add two in-context examples (ablation) | GPT-4V · II-Bench test | accuracy: 65.9 -> 67.7 | Helped |
| add three in-context examples (ablation) | Qwen-VL-MAX · II-Bench test | accuracy: 74.8 -> 53.6* | Hurt |
| evaluate image implication understanding instead of ordinary image perception | MLLMs versus humans · II-Bench test | accuracy: 74.8; 90.3 | Mixed |
Surprising findings
- Chain-of-thought prompting generally does not improve accuracy and can hurt smaller open-source models.
- Few-shot examples tend to reduce accuracy as shot count grows, e.g., Qwen-VL-MAX falls from 74.8% to 53.6% at 3-shot due to context length limits.
Limitations
- Subjective image interpretations may lead to inconsistent answers.
- Covers only specific domains and a portion of human knowledge.
- Evaluation metrics may not fully reflect sophisticated understanding.
Experimental setup
Datasets: II-Bench
Benchmarks: II-Bench
Metrics: accuracy miss rate error rate
Baselines: GPT-4V GPT-4o Gemini-1.5 Pro Qwen-VL-MAX LLaVA-1.6-34B InternVL-Chat-1.5 CogVLM2-Llama3-Chat MiniCPM-Llama3-2.5 Idefics2-8B Yi-VL-34B-Chat
Builds on: MMMU MMStar MMBench SEED
Closest papers by meaning
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.92)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.90)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.89)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.