Research map / Vision-language and multimodal / Visual question answering
The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition
Yuwen Tan, Yuan Qing, Boqing Gong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| LoRA finetune on iNat21-Plant hierarchical VQA tasks | Qwen2.5-VL-7B · iNat21-Plant | HCA: 17.67 -> 29.34 | Improved |
| Same LoRA finetuning evaluated on other datasets | Qwen2.5-VL-7B · ImgNet-Animal | HCA: 56.00 -> 58.62 | Improved |
| Same LoRA finetuning evaluated on CUB-200 | Qwen2.5-VL-7B · CUB-200 | HCA: 43.76 -> 46.17 | Improved |
| LoRA finetuning measured on text-only taxonomy QA of the LLM | LLM of Qwen2.5-VL-7B · iNat21-Plant (text-only) | text HCA: 64.21 -> 84.87 | Improved |
| Scale Qwen2.5-VL from 7B to 72B (ablation) | Qwen2.5-VL · iNat21-Animal | HCA: 19.43 -> 35.73 | Improved |
| Vary prompts (hierarchical, chain-of-thought, JSON taxonomy) (ablation) | VLLMs · CUB-200, iNat21-Plant, ImgNet-Animal | HCA / Acc_leaf | No clear effect |
| Linearly probe visual tokens instead of VLLM output (ablation) | Qwen2.5-VL-7B · CUB-200, iNat21-Plant | HCA | Improved |
| Compare original LLMs vs vision-language-tuned LLMs (ablation) | Qwen2.5-VL / InternVL2.5 LLMs · five taxonomies (text-only) | text HCA | Improved |
| Linear probes on LLM layer embeddings instead of LLM decoding (ablation) | Qwen2.5-VL-7B LLM · iNat21-Plant (text-only) | text HCA | Improved |
| Use closed-source LLM with strong taxonomy knowledge (GPT-4o) (ablation) | GPT-4o · five taxonomies | text HCA: 98.81 | Improved |
| Orthogonal hierarchical representation in LLM (ablation) | Qwen2.5-7B / Gemma · taxonomy QA | HCA | No clear effect |
| Reduce taxonomy depth (ablation) | LLMs of VLLMs · iNat21-Plant (text-only) | text HCA | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 0 of 1 on most shared results.
- BioCLIP2 — BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning (NeurIPS 2025) · better on 1 of 4 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR 2024 · similarity 0.88)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.88)
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.88)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.88)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.87)
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
