Research map / Vision-language and multimodal / Multimodal LLMs
A-Bench: Are LMMs Masters at Evaluating AI-generated Images?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces A-Bench, a diagnostic multiple-choice benchmark of 2,864 text-to-image outputs with expert-written questions, testing whether LMMs can answer semantic-alignment and quality-perception questions needed to evaluate AI-generated images; 23 LMMs fall well short of humans.
Problem
LMM-based evaluators for AI-generated images are used without verification of their basic semantic and quality perception abilities, and existing benchmarks rely mostly on natural images rather than AIGIs.
Approach
Instead of measuring correlation of LMM scores with human MOS, diagnose the underlying LMMs by checking whether they correctly answer fine-grained multiple-choice questions about AIGIs, split into high-level semantic understanding (A-Bench-P1) and low-level quality perception (A-Bench-P2), with human-expert-annotated questions.
Key results
- Best LMM Qwen2-VL-72B reaches 86.02% on A-Bench-P1 and 68.99% on A-Bench-P2, while the worst human scores 92.40% and 90.56%, a gap of about 14.62 points overall.
- Proprietary LMMs are roughly on par with the best open-source ones; Gemini 1.5 Pro scores 84.70% on P1 and 69.12% on P2.
- Most LMMs perform worst on generative distortion assessment, and all LMMs trail the poorest human on every subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase temperature from 0.0 to 0.5 (ablation) | DeepSeek-VL-7B · A-Bench overall | accuracy: 66.58±0.00 -> 65.11±1.72 | Hurt |
| Increase temperature from 0.5 to 1.0 (ablation) | DeepSeek-VL-7B · A-Bench overall | accuracy: 65.11±1.72 -> 62.04±4.51 | Hurt |
| Increase temperature from 0.0 to 0.5 (ablation) | LLaVA-NeXT-8B · A-Bench overall | accuracy: 67.75±0.00 -> 66.43±2.09 | Hurt |
| Increase temperature from 0.5 to 1.0 (ablation) | LLaVA-NeXT-8B · A-Bench overall | accuracy: 66.43±2.09 -> 63.77±3.86 | Hurt |
| Increase temperature from 0.0 to 0.5 (ablation) | LLaVA-v1.5-7B · A-Bench overall | accuracy: 62.97±0.00 -> 60.61±2.23 | Hurt |
| Increase temperature from 0.5 to 1.0 (ablation) | LLaVA-v1.5-7B · A-Bench overall | accuracy: 60.61±2.23 -> 59.22±4.01 | Hurt |
| Increase temperature from 0.0 to 0.5 (ablation) | Qwen-VL-7B · A-Bench overall | accuracy: 60.41±0.00 -> 58.17±1.89 | Hurt |
| Increase temperature from 0.5 to 1.0 (ablation) | Qwen-VL-7B · A-Bench overall | accuracy: 58.17±1.89 -> 55.22±6.04 | Hurt |
Surprising findings
- Proprietary models such as Gemini 1.5 Pro and GPT-4o struggle most on Contradiction Overcome, falling well below humans despite strong basic recognition.
- Human performance is more balanced across subcategories than LMM performance, which varies widely.
Limitations
- Benchmark may lag behind rapidly evolving generative models and LMMs.
- Full manual annotation with three-way review makes scaling up costly.
- Limited number of tested generators and evaluators may limit generalizability.
Experimental setup
Datasets: AIGIQA-20K
Benchmarks: A-Bench
Metrics: accuracy
Baselines: Human (best) Human (worst) random guess
Builds on: Q-Bench Q-Align AIGIQA-20K
Closest papers by meaning
- AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM (CVPR 2025 · similarity 0.88)
- Are High-Quality AI-Generated Images More Difficult for Models to Detect? (ICML 2025 · similarity 0.88)
- AI-generated Image Quality Assessment in Visual Communication (AAAI 2025 · similarity 0.87)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.87)
- GenAI Arena: An Open Evaluation Platform for Generative Models (NeurIPS 2024 · similarity 0.87)
- UVE: Are MLLMs Unified Evaluators for AI-Generated Videos? (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.