Research map / Vision-language and multimodal / Visual question answering
Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AutoConverter, a GPT-4o multi-agent pipeline that converts open-ended VQA questions into challenging multiple-choice questions, and uses it to build VMCBench (9,018 questions from 20 VQA datasets) for evaluating 33 VLMs.
Problem
Open-ended VQA evaluation is unreliable: rule-based metrics correlate poorly with true performance and model-based judges (GPT-4o) shift scores across versions, while writing good multiple-choice questions with plausible distractors is costly.
Approach
A multi-agent GPT-4o pipeline in which error-type-specific distractor proposers, a reviewer, and a selector increase difficulty, while a correctness evaluator and refiner iteratively remove ambiguous distractors, converting open-ended questions into correct, hard multiple-choice questions.
Key results
- Qwen2-VL-72B is the best model on VMCBench at 85.0% overall accuracy, above GPT-4o at 80.3%.
- Human annotation finds only 3% of score-5 AutoConverter questions on MMMU, MathVista, and AI2D are incorrect; VLMs score similar or lower on AutoConverter questions than on original ones.
- Rule-based metrics correlate only 0.09 (Spearman) with model-based human-proxy evaluation on VQAv2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Convert and refine questions with AutoConverter rather than naive single-prompt distractor generation | MMMU questions · MMMU | average model performance: 46.0 -> 40.7 | Helped |
| Add concept misunderstanding proposer (ablation) | AutoConverter · MMMU | average model performance: 43.8 -> 40.7 | Helped |
| Add reasoning error proposer (ablation) | AutoConverter · MMMU | average model performance: 43.5 -> 40.7 | Helped |
| Add visual misinterpretation proposer (ablation) | AutoConverter · MMMU | average model performance: 42.8 -> 40.7 | Helped |
| Add data processing error proposer (ablation) | AutoConverter · MMMU | average model performance: 42.7 -> 40.7 | Helped |
| Add question bias proposer (ablation) | AutoConverter · MMMU | average model performance: 44.2 -> 40.7 | Helped |
| Add reviewer agent for iterative distractor refinement (ablation) | AutoConverter · MMMU | average model performance: 42.7 -> 40.7 | Helped |
| Add correctness evaluator and question refiner (ablation) | AutoConverter · MMMU | correctness: 4.28 -> 4.69 | Helped |
| Add correctness evaluator and question refiner (ablation) | AutoConverter · MMMU | average model performance: 50.0 -> 40.7 | Mixed |
| Use multiple-choice evaluation instead of rule-based scoring of open-ended VQA answers | VLM evaluation on VQAv2 · VQAv2 | accuracy: 32% -> 88% | Helped |
| Evaluate open-ended answers with GPT-4o-0806 rather than GPT-4o-0513 | VLM evaluation on MMVet · MMVet | model-based evaluation score: 6% | Mixed |
| Scale model size from InstructBLIP-7B to Qwen2-VL-72B | VLMs evaluated on VMCBench · VMCBench Avg. | accuracy: 42.1% to 85.0% | Helped |
Surprising findings
- Model-based GPT-4o evaluation shifts absolute scores by about 6% between GPT-4o versions 0513 and 0806 despite perfect ranking correlation.
- Roughly 59% of correctness errors found in the converted benchmark stem from errors in the original ground-truth answers rather than from AutoConverter.
Limitations
- Around 2% of score-5 questions still contain errors per human annotation; the benchmark relies on original answers which can be wrong.
- Benchmark uses sampled subsets of up to 500 questions per dataset and a single prompt format for evaluation.
- Pipeline depends on proprietary GPT-4o, so reproducibility depends on API availability.
Experimental setup
Datasets: VQAv2 OKVQA MMVet VizWiz A-OKVQA MMStar SEEDBench MathVision GQA MMMU RealWorldQA MathVista
Benchmarks: VMCBench MMMU MathVista AI2D
Metrics: accuracy correctness score (1-5 Likert) Spearman correlation agreement with human evaluation
Baselines: naive distractor generation prompt rule-based metrics (EM, BLEU, ROUGE) model-based evaluation with GPT-4o
Builds on: agentic self-reflection role-playing agents
Closest papers by meaning
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.90)
- Right this way: Can VLMs Guide Us to See More to Answer Questions? (NeurIPS 2024 · similarity 0.90)
- Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR 2024 · similarity 0.90)
- Improving Automatic VQA Evaluation Using Large Language Models (AAAI 2024 · similarity 0.90)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.89)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.