Research map / Vision-language and multimodal / CLIP and zero-shot

Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Feature calibration for unseen-class detection · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PROMPT-CAM, which learns class-specific prompts on a frozen pre-trained ViT (DINO/DINOv2/BioCLIP) with a shared scoring vector so each class's multi-head attention maps reveal and localize fine-grained traits.

Problem

Saliency methods like Grad-CAM produce blurred, coarse heatmaps on ViTs that cover whole objects rather than the subtle traits distinguishing visually similar fine-grained categories such as bird species.

Approach

Add C learnable class-specific prompt tokens to a frozen ViT (a modified VPT-Deep where only the last layer's prompts are class-specific), score each class via inner products with a shared vector, and train with cross-entropy so each class's prompt attention maps localize its traits.

Key results

• PROMPT-CAM achieves insertion 0.61 and deletion 0.09 on CUB with DINO, better than Grad-CAM (0.52/0.17), Layer-CAM (0.54/0.13), Eigen-CAM (0.56/0.22) and attention roll-out (0.55/0.27).
• In a human study, participants recognized 60.49% of expert-identified traits with PROMPT-CAM versus 39.14% for TesNet and 30.39% for ProtoConcepts.
• Classification accuracy is slightly below linear probing with DINO (e.g., CUB 73.2% vs 78.6%).

Measured effects reported in the paper

Surprising findings

• Classification accuracy drops relative to linear probing, which the authors argue is intended since the model is constrained to rely on localized traits.
• Removing the red wing spot from a Red-winged Blackbird image causes the model to predict Boat-tailed Grackle, showing sensitivity to the identified trait.

Limitations

• Slight drop in classification accuracy versus linear probing.
• Misclassified images with occluded or unusual-pose traits are not explained well by the method.

Experimental setup

Datasets: CUB-200-2011 Birds-525 Stanford Dogs Oxford Pet iNaturalist-2021-Moths Fish Vista Rare Species Insects-2 iNaturalist-2021-Fungi Oxford Flowers Medicinal Leaf Stanford Cars

Benchmarks: CUB-200-2011 Birds-525 Stanford Dogs Oxford Pet iNaturalist-2021-Moths Fish Vista Rare Species Insects-2 iNaturalist-2021-Fungi Oxford Flowers Medicinal Leaf Stanford Cars

Metrics: insertion score deletion score classification accuracy trait recognition rate (human study)

Baselines: Grad-CAM Layer-CAM Eigen-CAM Attention roll-out ProtoPFormer TesNet ProtoConcepts INTR Linear Probing

Builds on: Visual Prompt Tuning INTR MCTformer Query2Label DINO

Closest papers by meaning

• Multi-scale Activation, Refinement, and Aggregation: Exploring Diverse Cues for Fine-Grained Bird Recognition (AAAI 2025 · similarity 0.88)
• Finer-CAM: Spotting the Difference Reveals Finer Details for Visual Explanation (CVPR 2025 · similarity 0.87)
• UniAP: Towards Universal Animal Perception in Vision via Few-Shot Learning (AAAI 2024 · similarity 0.85)
• Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images (CVPR 2025 · similarity 0.85)
• AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.85)
• Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.85)