Research map / Vision-language and multimodal / Multimodal LLMs

Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FG-BMK, a large fine-grained benchmark (1.01M questions, 0.28M images) that evaluates LVLMs via human-oriented VQA and machine-oriented retrieval/classification probes, revealing weaknesses in species-level recognition, attribute recognition, alignment effects, and perturbation robustness.

Problem

LVLMs are rarely evaluated on fine-grained image tasks (subordinate-category recognition and fine-grained feature discriminability), so their capability limits in this area are poorly understood.

Approach

A dual-perspective benchmark: human-oriented dialogue-style true/false, multiple-choice and short-answer questions built from 12 fine-grained datasets with hierarchical taxonomy labels, plus machine-oriented retrieval and classification using visual features (following DINOv2's protocol), used to probe training paradigm, alignment granularity, and perturbation robustness.

Key results

• InternVL3 accuracy on species-level true/false questions falls to 62.48% versus 99.77% at class level on CUB-200-2011.
• Perturbing EVA-CLIP visual features drops CUB-200-2011 classification from 88.95% to 24.94%, versus 93.05% to 50.76% on CIFAR-100.
• Aligning LLaVA visual features to text lowers classification accuracy by an average of 3.39% relative to original features.

Surprising findings

• Fine-grained categories are almost absent from the LVLM training data, suggesting knowledge bias is inherited from the underlying LLM.
• A smaller DINOv2-B outperforms the larger BEiT3-L by 8.08% on CUB-200-2011 in classification.

Limitations

• Machine-oriented evaluation uses earlier model versions rather than the latest LVLMs.
• Granularity-matched alignment data construction may still contain residual mismatch.

Experimental setup

Datasets: CUB-200-2011 Stanford Dogs Stanford Cars Food-101 FGVC Aircraft Flowers102 DeepFashion iNat2021 Products-10K VegFru SkinCon Wine

Benchmarks: FG-BMK CUB-200-2011 CIFAR-100

Metrics: accuracy mAP Top-1 accuracy

Baselines: DINOv2 EVA-CLIP CoCa BEiT3 BLIP-2 CAP

Builds on: DINOv2 LLaVA

Closest papers by meaning

• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.93)
• MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.93)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.93)
• Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.91)
• Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.91)
• GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs (ICLR 2026 · similarity 0.91)