Research map / Vision-language and multimodal / Multimodal LLMs

AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AVA-Bench, a benchmark that disentangles 14 atomic visual abilities to evaluate vision foundation models via per-ability LLaVA-style probes, and shows a 0.5B LLM gives similar VFM rankings to a 7B LLM at about 8x lower cost.

Problem

Generic LLM-based VQA evaluation of vision foundation models conflates instruction-tuning data mismatch and multiple required visual abilities, so failures cannot be attributed to specific visual shortcomings.

Approach

Decouple VFM evaluation into 14 atomic visual abilities, each with distribution-matched train/test splits built from 26 datasets, and fine-tune a connector plus LoRA per ability on top of a frozen VFM with an LLM head; also show a 0.5B LLM preserves VFM rankings at much lower cost.

Key results

• SigLIP-1/2 and AIMv2 achieve the highest average rank across all AVAs, highlighting the value of language supervision.
• A 0.5B Qwen2 LLM yields VFM rankings similar to a 7B Vicuna-1.5 while cutting evaluation GPU hours by about 8x.
• DINOv2 performs comparably or better than language-supervised models on localization, absolute depth, and orientation.

Surprising findings

• Linear-probe fine-grained recognition accuracy on DINOv2 drops from 66.3% pre-connector to 25.67% post-connector, suggesting information loss at modality alignment.
• Almost all VFMs, including weaker overall ones, excel in at least one AVA (e.g., SAM in color).

Limitations

• Frozen-VFM plus LLM-head protocol may conflate alignment loss with visual deficiency, as shown by the post-connector accuracy drop.
• Per-AVA probes use isolated, simplified questions that may not reflect composite real-world VQA.

Experimental setup

Datasets: Objects365 LVIS iNaturalist-2021 DIOR VQAv2 FSC-147 CARPK CrowdHuman NYU-Depth V2 KITTI CUB-200-2011 FGVC-Aircraft

Benchmarks: AVA-Bench

Metrics: GIoU normalized MAE CIEDE2000 ANLS accuracy

Baselines: DINOv2 CLIP SigLIP-1 SigLIP-2 AIMv2 SAM MiDaS RADIO

Builds on: LLaVA LoRA

Closest papers by meaning

• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.90)
• Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.90)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.89)
• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.89)
• Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.89)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)