Research map / Vision-language and multimodal / Multimodal LLMs
Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finedefics improves fine-grained visual recognition in MLLMs by aligning visual objects and category names via attribute-description contrastive learning with hard negatives, followed by classification-centered instruction tuning on Idefics2.
Problem
MLLMs underperform on fine-grained visual recognition (identifying subordinate-level categories), which hurts object-centric VQA and reasoning; the paper attributes the root cause to misalignment between visual object representations and category names in the LLM representation space.
Approach
Use LLM/VQA-generated per-sample attribute descriptions as an intermediate anchor, and apply contrastive learning on object-attribute and attribute-category pairs (with hard negatives from similar categories mined by CLIP) before instruction tuning on FGVR data.
Key results
- Finedefics achieves 76.84% average accuracy over six FGVR datasets, versus 65.95% for Idefics2 and 67.41% for Qwen-VL-Chat (+10.89 and +9.43 points).
- Ablation: contrastive learning on object-category only reaches 72.72 avg, while object-attribute-category triples reach 76.84.
- Applying attribute augmented alignment to LLaVA 1.5 raises its average from 43.24 to 57.21.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Idefics2 with attribute-augmented alignment and classification-centered instruction tuning | Idefics2 · FGVR six-dataset average | accuracy: 67.41 -> 76.84 | Helped |
| Add attribute descriptions to object-category contrastive learning (ablation) | Idefics2 · FGVR six-dataset average | accuracy: 72.72 -> 76.84 | Helped |
| Combine contrastive learning and instruction tuning in one stage instead of two (ablation) | Idefics2 · FGVR six-dataset average | accuracy: 76.84 -> 25.42 | Hurt |
| Use hard negatives rather than randomly sampled simple negatives (ablation) | Idefics2 contrastive learning · FGVR six-dataset average | accuracy: 74.26 -> 76.84 | Helped |
| Remove instruction tuning and use pretraining alone (ablation) | Idefics2 · FGVR six-dataset average | accuracy: 76.84 -> 0.00 | Hurt |
| Remove pretraining and use instruction tuning alone (ablation) | Idefics2 · FGVR six-dataset average | accuracy: 76.84 -> 76.13 | Hurt |
| Use complete rather than noisy attribute descriptions (ablation) | Idefics2 · FGVR six-dataset average | accuracy: 75.62 -> 76.84 | Helped |
| Use long rather than short attribute descriptions (ablation) | Idefics2 · FGVR six-dataset average | accuracy: 76.11 -> 76.84 | Helped |
| Apply attribute augmented alignment | LLaVA 1.5 · Dog-120 | accuracy: 38.96 -> 57.10 | Helped |
| Train on FGVR tasks | Idefics2 · ImageNet-adversarial | accuracy: 79.84 -> 75.96 | Hurt |
Surprising findings
- Naive fine-tuning of Idefics2 on open-set FGVR data collapses performance to 0.03% average accuracy, showing instruction-following degradation.
- Finedefics trained only on FGVR data loses about 1.6 points on ImageNet-adversarial relative to Idefics2 (75.96 vs 79.84), indicating some forgetting of general recognition.
Limitations
- Struggles to learn new subordinate-level categories (continual learning not addressed).
- Performance on common object recognition (ImageNet variants) drops slightly after FGVR-only training.
- Relies on GPT-4/LLM and VQA models to generate attribute descriptions, which may be noisy.
Experimental setup
Datasets: CaltechUCSD Bird-200 Stanford Car-196 Stanford Dog-120 Flower-102 Oxford-IIIT Pet-37 FGVC-Aircraft ImageNet-adversarial ImageNet-rendition ImageNet-sketch
Benchmarks: Dog-120 Bird-200 Aircraft-102 Flower-102 Pet-37 Car-196 IN-adversarial IN-rendition IN-sketch
Metrics: classification accuracy average accuracy clustering accuracy cosine similarity alignment
Baselines: LLaVA 1.5 LLaVA-Next MobileVLM v2 InstructBLIP Phi-3-Vision BLIP2 InternLM XComposer 2 Pali-Gemma Idefics1 Idefics2 Qwen-VL-Chat FineR
Builds on: Idefics2 CLIP FineR QLoRA
Closest papers by meaning
- Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.93)
- Democratizing Fine-grained Visual Recognition with Large Language Models (ICLR 2024 · similarity 0.92)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.91)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.90)
- GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs (ICLR 2026 · similarity 0.89)
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.