Research map / Vision-language and multimodal / CLIP and zero-shot
Why are Visually-Grounded Language Models Bad at Image Classification?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that visually-grounded language models lag far behind CLIP on image classification, traces the gap mainly to training data coverage of classes rather than inference, objective, or information loss, and shows adding classification data improves VLM accuracy and ImageWikiQA performance.
Problem
Visually-grounded language models (VLMs) such as GPT-4V, Gemini, Claude and LLaVA perform far worse than CLIP on standard image classification benchmarks despite using CLIP encoders and many more parameters; the cause of this gap is unclear.
Approach
Systematically tests inference, training, and data hypotheses and finds the gap is primarily data-driven: class information is preserved in the VLM latent space but only decodable with sufficient class exposure in training data. Integrating ImageNet classification data into instruction tuning then improves both classification and general VQA-style capabilities.
Key results
- On ImageNet, GPT-4 reaches 60.6% closed-world accuracy versus 79.2% for EVA-G, and the best open-world public VLM (LLaVA-NeXT-M7B) reaches only 32.3%.
- LLaVA-1.5-7B fine-tuned on the ImageNet classification data reaches 85.7% ImageNet accuracy, matching CLIP-L fine-tuned (85.2%), and Spearman correlation between class frequency in training data and accuracy is 0.76.
- Fine-tuning LLaVA-1.5-7B on ImageNet plus LLaVA instruction data improves ImageWikiQA accuracy from 38.0% to 49.8% (+11.8%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA1.5-7B on classification data | LLaVA1.5-7B · ImageNet | accuracy: 22.8 -> 84.4 | Helped |
| Add LLaVA instruction-tuning data to ImageNet fine-tuning (ablation) | LLaVA1.5-7B fine-tuned on ImageNet · ImageWikiQA | accuracy: 30.6 -> 49.8 | Helped |
| Fine-tune on ImageNet classification data only (ablation) | LLaVA1.5-7B · ImageWikiQA | accuracy: 38.0 -> 30.6 | Hurt |
| Use caption-focused instead of classification-focused fine-tuning data (ablation) | LLaVA1.5-7B · Flowers102, StanfordCars, and Caltech101 | accuracy: 97.6 -> 92.0; 90.4 -> 85.4; 97.5 -> 95.7 | Mixed |
| Use probabilistic inference with classifier-free guidance instead of direct generation (ablation) | LLaVA1.5-7B · ImageNet | accuracy: 22.7% -> 47.6% | Helped |
| Use chain-of-thought prompting (ablation) | LLaVA1.5-7B · ImageNet, closed-world | accuracy: 10.2 -> 18.1 | Helped |
| Use chain-of-thought prompting (ablation) | BLIP2-2.7B · ImageNet, closed-world | accuracy: 14.2 -> 8.5 | Hurt |
| Replace direct generation with probabilistic inference (ablation) | BLIP2-2.7B · StanfordCars | accuracy: 0.0 -> 48.8 | Mixed |
| Fine-tune only the projector rather than projector plus LM with LoRA (ablation) | LLaVA1.5-7B · ImageNet, Flowers102, StanfordCars, and Caltech101 | accuracy: 85.7 vs. NaN; 97.6 vs. 97.9; 90.4 vs. 90.7; 97.5 vs. 97.8 | No clear effect |
| Increase class exposure in VLM training data | LLaVA1.5-7B · ImageNet classes | accuracy: 3.0% -> 82.7% | Helped |
| Add label synonyms during evaluation (ablation) | VLMs · ImageNet, open-world | accuracy: 22.8 -> 24.6; 53.6 -> 56.3 | Helped |
| Provide a smaller candidate label set (ablation) | LLaVA1.5-7B · ImageNet, closed-world | accuracy: 46.6 -> 94.3 | Helped |
Surprising findings
- Fine-tuning LLaVA-1.5-7B only on ImageNet lowers ImageWikiQA accuracy from 38.0% to 30.6%, indicating catastrophic forgetting of general capabilities.
- Caption-focused fine-tuning data achieves nearly the same classification accuracy as classification-formatted data, suggesting data presence matters more than data type.
Limitations
- Only two VLM architectures (LLaVA and BLIP) are analyzed in depth, and proprietary VLM training data is unavailable for data-side analysis.
- The proposed fix requires extensive training on large classification datasets; zero-shot decoding of latent information is not addressed.
- Fine-tuning solely on classification data causes catastrophic forgetting of general capabilities.
Experimental setup
Datasets: ImageNet Flowers102 StanfordCars Caltech101 LLaVA-1.5 instruction-tuning data
Benchmarks: ImageNet Flowers102 StanfordCars Caltech101 ImageWikiQA
Metrics: accuracy probing accuracy Spearman correlation
Baselines: GPT-4 Gemini Pro Claude 3 Opus LLaVA-1.5 LLaVA-NeXT BLIP-2 InstructBLIP CLIP-L EVA-G
Builds on: LLaVA InstructBLIP CLIP LoRA
Closest papers by meaning
- Does VLM Classification Benefit from LLM Description Semantics? (AAAI 2025 · similarity 0.90)
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.90)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.88)
- Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · similarity 0.88)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.