Research map / Vision-language and multimodal / CLIP and zero-shot
Delving into Multimodal Prompting for Fine-Grained Visual Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MP-FGVC, which adapts a pre-trained CLIP model to fine-grained visual classification using subcategory-specific vision prompts (token selection), learnable discrepancy-aware text prompts, and a vision-language fusion module, trained in two stages.
Problem
Fine-grained visual classification requires capturing subtle subcategory discrepancies, but existing CLIP prompt strategies focus on supercategory semantics and FGVC datasets lack textual annotations, so it is unclear how to adapt vision-language models to close-set FGVC.
Approach
Combine a parameter-free top-k attention-based token selection in the penultimate ViT layer (SsVP) with learnable text tokens paired with the supercategory name (DaTP), and fuse them with a cross-modal attention module (VLFM), trained in two stages with the CLIP text encoder frozen first.
Key results
- MP-FGVC reaches 91.8% on CUB-200-2011 and 91.0% on Stanford Dogs, the best among compared CNN and ViT methods reported.
- On NABirds, MP-FGVC reaches 91.0%, versus 90.5% for the reproduced TransFG.
- On Food101, MP-FGVC reaches 93.0%, versus 92.1% for the reproduced TransFG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the full MP-FGVC multimodal prompting and adaptation method | ViT-B-16 · CUB-200-2011 | Top-1 accuracy: 91.3 -> 91.8 | Helped |
| Add the full MP-FGVC multimodal prompting and adaptation method | ViT-B-16 · Stanford Dogs | Top-1 accuracy: 90.7 -> 91.0 | Helped |
| Add the full MP-FGVC multimodal prompting and adaptation method | ViT-B-16 · NABirds | Top-1 accuracy: 90.5 -> 91.0 | Helped |
| Add the full MP-FGVC multimodal prompting and adaptation method | ViT-B-16 · Food101 | Top-1 accuracy: 92.1 -> 93.0 | Helped |
| Add DaTP to the baseline (ablation) | ViT baseline · CUB-200-2011 | accuracy: 90.8 -> 91.3 | Helped |
| Add DaTP to the baseline (ablation) | ViT baseline · NABirds | accuracy: 90.0 -> 90.4 | Helped |
| Add SsVP to the baseline (ablation) | ViT baseline · CUB-200-2011 | accuracy: 90.8 -> 91.2 | Helped |
| Add SsVP to the baseline (ablation) | ViT baseline · NABirds | accuracy: 90.0 -> 90.3 | Helped |
| Add VLFM after Baseline + DaTP (ablation) | MP-FGVC prompts · CUB-200-2011 | accuracy: 91.3 -> 91.5 | Helped |
| Add VLFM after Baseline + DaTP (ablation) | MP-FGVC prompts · NABirds | accuracy: 90.4 -> 90.6 | Helped |
| Use learnable discrepancy tokens with the supercategory name instead of a handcrafted class-name prompt (ablation) | MP-FGVC · CUB-200-2011 | accuracy: 91.5 -> 91.8 | Helped |
| Use learnable discrepancy tokens with the supercategory name instead of a handcrafted class-name prompt (ablation) | MP-FGVC · NABirds | accuracy: 90.7 -> 91.0 | Helped |
Surprising findings
- Learnable text tokens with the supercategory name outperformed handcrafted 'a photo of a [class]' prompts, and the paper reports performance degrades when the subcategory-level prompt length exceeds 32 tokens.
Limitations
- Improvements over the best baselines are small (0.3-0.9 points).
- Relies on the supercategory name and frozen CLIP text encoder; performance saturates or drops when the number of selected tokens or text tokens grows.
Experimental setup
Datasets: CUB-200-2011 Stanford Dogs NABirds Food101
Benchmarks: CUB-200-2011 Stanford Dogs NABirds Food101
Metrics: Top-1 accuracy
Baselines: TransFG IELT SIM-Trans API-Net CAL PRIS MSMVFA PRENet PSM MHVM
Builds on: CLIP ViT TransFG CoOp
Closest papers by meaning
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.90)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.89)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.89)
- Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
- Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.88)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.