Research map / Vision-language and multimodal / Multimodal LLMs
MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MIA-Bench, a 400-prompt image-instruction benchmark scored by GPT-4o as judge for strict multimodal instruction adherence, evaluates 29 MLLMs, and shows SFT on 5k self-generated diverse instruction data improves LLaVA-NeXT-13b's score by about 10 points.
Problem
Existing multimodal benchmarks either use fixed-form VQA or loosely judged open-ended chat, so the precise adherence of MLLMs to complex, compositional instructions (length, format, grammar, mention constraints) is under-measured.
Approach
A benchmark of 400 hand-written image-prompt pairs with layered sub-instructions, where a judge model (GPT-4o) assigns weighted per-sub-instruction scores, plus an SFT study using GPT-4v-generated diverse instruction-tuning data to improve adherence.
Key results
- GPT-4o achieves the top MIA-Bench score of 88.58, ahead of GPT-4v (86.11) and Claude-3-Opus (84.50); InternVL-Chat-v1.5 ranks lowest among five top models despite leading on other meta-rankings.
- SFT of LLaVA-NeXT-13b on 5k GPT-4v-generated diverse instruction data raises MIA-Bench score from 69.16 to 78.85 (about 10 points), with minor regressions on other benchmarks such as TextVQA (64.26 to 63.20).
- Rankings of the top models remain stable across several GPT-4o variant judges and Claude-3-Opus as judge, where GPT-4o still scores highest.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA-NeXT-13b on DIT data (ablation) | LLaVA-NeXT-13b · MIA-Bench | Meta-Avg: 69.16 -> 78.85 | Helped |
| Add LLaVA-Instruct150k to DIT fine-tuning data (ablation) | LLaVA-NeXT-13b + DIT · MIA-Bench | Meta-Avg: 78.85 -> 78.90 | No clear effect |
| Fine-tune LLaVA-NeXT-13b on DIT data (ablation) | LLaVA-NeXT-13b · MMBench | score: 70.6 -> 68.6 | Hurt |
| Fine-tune LLaVA-NeXT-13b on DIT data (ablation) | LLaVA-NeXT-13b · TextVQA | score: 64.26 -> 63.20 | Hurt |
| Fine-tune LLaVA-NeXT-13b on DIT data (ablation) | LLaVA-NeXT-13b · VQA2 | score: 82.80 -> 82.58 | No clear effect |
| Fine-tune LLaVA-NeXT-13b on DIT data (ablation) | LLaVA-NeXT-13b · POPE | score: 87.7 -> 86.9 | Hurt |
| Fine-tune LLaVA-NeXT-13b on DIT data (ablation) | LLaVA-NeXT-13b · VizWiz | score: 60.41 -> 59.72 | Hurt |
| Increase LLaVA-NeXT model size | LLaVA-NeXT · MIA-Bench | Meta-Avg: LLaVA-NeXT-13b: 69.16; LLaVA-NeXT-34b: 75.61; LLaVA-NeXT-110b: 79.84 | Helped |
Surprising findings
- InternVL-Chat-v1.5, which has the highest meta-ranking on other benchmarks among five top models, ranks lowest on MIA-Bench.
- Adding LLaVA-Instruct150k to the DIT data did not significantly improve MIA-Bench and reduced several other benchmark scores (e.g., TextVQA 54.24).
Limitations
- Real-world instruction space is effectively infinite, so the 400 prompts cover only a subset of possible challenges.
- Scoring depends on a GPT-4o judge, with about 1% variation across runs and possible judge bias.
- SFT experiments are limited to a single backbone (LLaVA-NeXT-13b) and a small dataset.
Experimental setup
Datasets: COCO 2017 SBU TextVQA Flickr LLaVA-Instruct150k
Benchmarks: MIA-Bench MMBench TextVQA VQA2 LLaVA-itw POPE VizWiz IFEval MME MMMU MM-Vet HallusionBench
Metrics: weighted sub-instruction score (percentage) ranking correlation
Baselines: GPT-4v Claude-3-Opus Gemini-1.0-Pro Reka-Core InternVL-Chat-v1.5 LLaVA-NeXT-13b MiniCPM-Llama3-v2.5 CogVLM2
Builds on: LLaVA-NeXT IFEval InfoBench LLaVA-Instruct150k
Closest papers by meaning
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · similarity 0.91)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.91)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.90)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.90)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.