Research map / Vision-language and multimodal / Multimodal LLMs
ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ConMe, a compositional reasoning benchmark built by a multi-stage pipeline in which GPT-4V generates hard binary CR questions informed by open VLMs' descriptions and answers, filtered by whether open VLMs fail, yielding large accuracy drops for modern VLMs.
Problem
Modern VLMs with LLM decoders reportedly perform well on existing compositional reasoning benchmarks, which use LLM-only negative generation that yields out-of-distribution or image-implausible negatives, so these benchmarks may not be challenging enough.
Approach
Use VLMs (GPT-4V plus open VLMs) in a multi-turn 'conversation' to generate CR questions grounded in image context, iteratively filtering out samples that all weaker VLMs answer correctly, so the benchmark targets what current VLMs miss.
Key results
- Across 7 VLMs, average accuracy drops from 88.6% on SugarCrepe to 65.3% on ConMe (23.3 points on average).
- Seen models LLaVA 1.6-7b and InstructBLIP Flan-T5 drop by up to 31.7% and 33.0% respectively on ConMe versus SugarCrepe.
- GPT-4V accuracy falls from 91.2% on SugarCrepe to 80.1% on ConMe; unseen InternLM-XComposer2-VL-7b and Idefics2-8b drop by 12.3 and 15.4 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on ConMe instead of SugarCrepe | LLaVA 1.5-7b · ConMe vs SugarCrepe | accuracy (%): 88.5 -> 57.7 | Hurt |
| Evaluate on ConMe instead of SugarCrepe | LLaVA 1.6-7b · ConMe vs SugarCrepe | accuracy (%): 89.2 -> 57.5 | Hurt |
| Evaluate on ConMe instead of SugarCrepe | InstructBLIP Flan-T5 · ConMe vs SugarCrepe | accuracy (%): 91.4 -> 58.5 | Hurt |
| Evaluate on ConMe instead of SugarCrepe | InstructBLIP Vicuna-7b · ConMe vs SugarCrepe | accuracy (%): 82.5 -> 53.6 | Hurt |
| Evaluate on ConMe instead of SugarCrepe | InternLM-XComposer2-VL-7b · ConMe vs SugarCrepe | accuracy (%): 92.0 -> 79.7 | Hurt |
| Evaluate on ConMe instead of SugarCrepe | Idefics2-8b · ConMe vs SugarCrepe | accuracy (%): 85.5 -> 70.1 | Hurt |
| Evaluate on ConMe instead of SugarCrepe | GPT-4V · ConMe vs SugarCrepe | accuracy (%): 91.2 -> 80.1 | Hurt |
| Add filtering after Iteration 1 (ablation) | ConMe curation evaluated with four seen VLMs · ConMe | accuracy (%): ~30% | Hurt |
| Add the second iteration of GPT-4V question generation (ablation) | ConMe curation evaluated with four seen VLMs · ConMe | accuracy (%) | Hurt |
| Use perplexity inference instead of generate inference (ablation) | ConMe evaluation · ConMe | accuracy (%): ~2% | No clear effect |
| Use the manually verified subset instead of all ConMe samples (ablation) | VLM evaluations · Manual Subset vs ConMe | accuracy (%): 0.5% | No clear effect |
| Improve LLaVA from version 1.5 to 1.6 | LLaVA · ConMe human-filtered partition, emotion errors | mistake rate (%): 40% to 27% | Helped |
Surprising findings
- GPT-4V is 'fooled' by its own generated questions, with an 11.2-point drop on ConMe.
- Manual verification found about 20% erroneous samples, yet the accuracy on the verified 1000-sample subset is nearly identical to the full benchmark.
Limitations
- Dataset quality depends on the GPT-4V generator, which introduces errors (about 20% in manual check).
- Filtering against weaker VLMs may bias the benchmark toward their specific failure modes.
- Benchmark built on only 919 SugarCrepe COCO images.
Experimental setup
Datasets: SugarCrepe MS-COCO validation
Benchmarks: ConMe SugarCrepe
Metrics: accuracy performance drop
Baselines: SugarCrepe LLaVA 1.5-7b LLaVA 1.6-7b InstructBLIP Flan-T5 InstructBLIP Vicuna-7b InternLM-XComposer2-VL-7b Idefics2-8b GPT-4V
Builds on: SugarCrepe VQAScore
Closest papers by meaning
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.92)
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.91)
- Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.91)
- Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.90)
- CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.90)
- Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.