Research map / Vision-language and multimodal / Multimodal LLMs
Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Q-Bench, a zero-shot benchmark evaluating MLLMs on low-level vision via perception QA (LLVisionQA), expert-written long descriptions (LLDescribe) with GPT-judged scoring, and IQA score prediction using a softmax over good/poor token logits.
Problem
Existing MLLM benchmarks focus on high-level semantics, leaving MLLMs' low-level visual perception, description, and quality assessment abilities unmeasured.
Approach
Three-realm benchmark covering low-level perception (multi-choice QA over 2,990 images), description (golden expert text evaluated by GPT on completeness, precision, relevance), and assessment (IQA correlation using a softmax over good/poor logits to yield quantifiable scores).
Key results
- InternLM-XComposer-VL achieves 64.35% overall on LLVisionQA test, versus 73.36% for GPT-4V and 81.74% for senior-level humans.
- On IQA, InternLM-XComposer-VL reaches average SRCC/PLCC of 0.541/0.581 across 7 datasets, about 20% above CLIP-ViT-Large-14 (0.354/0.368).
- GPT-4V outperforms the best open-source MLLM by about 9 points on LLVisionQA perception accuracy and matches junior-level humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use perplexity-based close-set inference instead of normal multi-choice inference (ablation) | Kosmos-2 · LLVisionQA | overall accuracy: 41.47% -> 47.26% | Helped |
| Use five-round GPT voting instead of single-round evaluation | GPT evaluation of LLVisionQA answers · Human-checked sample set | GPT evaluation accuracy: 93.2% to 98.4% | Helped |
| Use softmax pooling of good/poor token logits instead of argmax token outputs | MLLM image quality assessment · Various IQA datasets | correlation with human ratings | Helped |
| Use the high/low token pair instead of good/poor for InstructBLIP | InstructBLIP (Flan-T5-XL) · Majority of IQA databases | IQA performance | Helped |
| Use multi-scale visual input features | LLVisionQA perception models · LLVisionQA test | accuracy on distortions versus other low-level attributes | Mixed |
| Use InternLM-XComposer-VL instead of CLIP-ViT-Large-14 | Image quality assessment · Assessment average | SRCC/PLCC: 20% | Helped |
Surprising findings
- Most MLLMs perform worse on distortions than on other low-level attributes.
- MLLMs show strong bias toward answering 'yes' on yes-or-no questions.
Limitations
- Test subset kept private and dev subset release is future, limiting open reproducibility.
- GPT-based judging introduces variability and reliance on a closed model.
- MLLMs show prompt-format failures (e.g., Kosmos-2) that confound multi-choice results.
Experimental setup
Datasets: KONiQ-10K SPAQ LIVE-FB LIVE-itw CGIQA-6K AGIQA-3K KADID-10K ImageRewardDB LIVEMultiDistortion Corrupted COCO
Benchmarks: LLVisionQA LLDescribe KONiQ-10K SPAQ LIVE-FB LIVE-itw CGIQA-6K AGIQA-3K KADID-10K
Metrics: accuracy completeness/precision/relevance score SRCC PLCC
Baselines: GPT-4V NIQE CLIP-ViT-Large-14 Kosmos-2 Otter-v1 mPLUG-Owl InstructBLIP
Builds on: LLaVA CLIP Shikra
Closest papers by meaning
- Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models (CVPR 2024 · similarity 0.95)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.92)
- MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs (ICML 2026 · similarity 0.91)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.91)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
- II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.