Research map / Vision-language and multimodal / Human and embodied modeling
Democratizing Fine-grained Visual Recognition with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FineR, a training-free pipeline that uses BLIP-2 to extract part-level visual attributes from a few unlabelled images, ChatGPT to reason about fine-grained class names, and CLIP to build a classifier for vocabulary-free fine-grained recognition.
Problem
Fine-grained visual recognition normally needs expert annotations (part labels, attributes, descriptions). The paper targets recognizing subordinate-level categories with only a few unlabelled images per class and no expert knowledge or label vocabulary.
Approach
Use a VQA model to turn images into textual attribute descriptions, prompt an LLM (acting as an expert proxy) to reason about candidate subordinate class names from those descriptions, denoise names with CLIP, and build a fused text-image CLIP classifier for zero-shot inference.
Key results
- FineR averages 57.0% cACC and 64.3% sACC over five fine-grained datasets with 3 images per class, versus 47.2% and 58.6% for the second-best BLIP-2.
- On the new Pokemon dataset FineR discovers 7/10 ground-truth categories, outperforming the second-best method by +31.6% cACC and +25.9% sACC.
- With imbalanced discovery sets FineR averages 51.9% cACC and 61.2% sACC, still best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FineR instead of BLIP-2 | FGVR on five standard datasets with 3 discovery images per category · Five FGVR datasets, average | cACC: +9.8% | Helped |
| Use FineR instead of BLIP-2 | FGVR on five standard datasets with 3 discovery images per category · Five FGVR datasets, average | sACC: +5.7% | Helped |
| Use FineR instead of BLIP-2 | FGVR with an imbalanced discovery set · Five FGVR datasets, average | cACC: +7.3% | Helped |
| Use FineR instead of BLIP-2 | FGVR with an imbalanced discovery set · Five FGVR datasets, average | sACC: +2.2% | Helped |
| Add noisy name denoising (ablation) | FineR · Five FGVR datasets, average | cACC: +6.0% | Helped |
| Add noisy name denoising (ablation) | FineR · Five FGVR datasets, average | sACC: +4.7% | Helped |
| Add the multimodal classifier (ablation) | FineR with noisy name denoising · Five FGVR datasets, average | cACC: 53.5 -> 53.7 | Helped |
| Add image augmentations with K=10 (ablation) | FineR with NND and MMC · Five FGVR datasets, average | sACC: 63.3 -> 64.3 | Helped |
| Add the multimodal classifier (ablation) | FineR with noisy name denoising · Dog-120 | cACC: 50.8 -> 39.4 | Hurt |
| Add the multimodal classifier (ablation) | FineR with noisy name denoising · Car-196 and Pet-37 | cACC: 47.4 -> 45.7; 69.1 -> 65.4 | Mixed |
| Add image augmentations with K=10 (ablation) | FineR with NND and MMC · Five FGVR datasets, average | cACC: 56.9 -> 57.0 | Helped |
Surprising findings
- The WordNet baseline (CLIP over a 119k noun vocabulary) fails to discover any Pokemon categories (0/10), and CaSED mostly predicts real-world analogs of Pokemon characters.
- Multi-modal classifier fusion slightly hurts on Dog-120 (cACC drops from 50.8 to 39.4 when adding MMC without augmentation), attributed to visual bias in the few discovery images.
Limitations
- Performance depends on the VQA model's ability to describe attributes and on LLM knowledge; VQA models struggle with fine-grained categories.
- Visual bias from few discovery images can hurt the vision-based classifier on some datasets (e.g., Dog-120).
- Relies on ChatGPT via API, which is closed-source; open-source replacement (Vicuna-13B) is only a generalizability check.
Experimental setup
Datasets: Caltech-UCSD Bird-200 Stanford Car-196 Stanford Dog-120 Flower-102 Oxford-IIIT Pet-37
Benchmarks: Caltech-UCSD Bird-200 Stanford Car-196 Stanford Dog-120 Flower-102 Oxford-IIIT Pet-37 Pokemon (new)
Metrics: clustering accuracy (cACC) semantic similarity accuracy (sACC)
Baselines: CLIP zero-shot (ground-truth names upper bound) WordNet baseline BLIP-2 CLEVER SCD CaSED KMeans Sinkhorn-Knopp (CLIP and DINO)
Builds on: BLIP-2 CLIP chain-of-thought prompting CaSED SCD
Closest papers by meaning
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.92)
- Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.92)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
- GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.88)
- The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding (CVPR 2024 · similarity 0.87)
- GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.