Research map / Architectures and efficiency / CNN design
Interpretable Image Classification with Adaptive Prototype-based Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProtoViT, a prototype-based interpretable image classifier using ViT backbones with deformable, adaptively sized prototypes built from greedy adjacency-masked patch matching, reporting higher CUB and Cars accuracy than prior prototype models.
Problem
Existing prototype-based interpretable classifiers rely on CNN backbones and spatially rigid prototypes that cannot handle geometric variation, and prototype-based ViT models lack well-defined visual cases (no projection to real patches).
Approach
Each prototype is a set of K sub-prototypes matched greedily, without replacement, to geometrically adjacent ViT patch tokens (adjacency mask), with learnable slot indicators that prune sub-prototypes; a coherence loss keeps sub-prototypes semantically similar, and class-token-subtracted features are used so prototypes can be projected to exact image patches.
Key results
- ProtoViT (DeiT-Small, K=4, r=1) reaches 85.37% accuracy on CUB-200-2011 and 91.84% on Cars, above ProtoPFormer (84.85%) and ViT-Net (84.26%) with the same backbone.
- With CaiT-XXS-24, ProtoViT achieves 85.82% on CUB and 92.40% on Cars, versus 84.79% and 91.04% for ProtoPFormer.
- On the Location Misalignment Benchmark, ProtoViT (DeiT-Small) reports PRC 1.28 and PAC 2.92, lower than ProtoPNet, TesNet, ProtoPool and ProtoTree on those metrics, though its PLC (21.68) is lower than ProtoPNet (24.0) and ProtoPool (31.8) but higher than TesNet (16.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProtoViT instead of ProtoPFormer | DeiT-Tiny prototype-based classifier · CUB-200-2011 | accuracy: 82.26 -> 82.92 $\pm$ 0.5 | Helped |
| Use ProtoViT instead of ProtoPFormer | DeiT-Tiny prototype-based classifier · Stanford Cars | accuracy: 88.48 -> 89.02 $\pm$ 0.1 | Helped |
| Use ProtoViT instead of ProtoPFormer | DeiT-Small prototype-based classifier · CUB-200-2011 | accuracy: 84.85 -> 85.37 $\pm$ 0.13 | Helped |
| Use ProtoViT instead of baseline | DeiT-Small classifier · Stanford Cars | accuracy: 90.06 -> 91.84 $\pm$ 0.3 | Helped |
| Use ProtoViT instead of ProtoPFormer | CaiT-XXS 24 prototype-based classifier · CUB-200-2011 | accuracy: 84.79 -> 85.82 $\pm$ 0.15 | Helped |
| Use ProtoViT instead of TesNet | CaiT-XXS 24 prototype-based classifier · Stanford Cars | accuracy: 92.6 $\pm$ 0.3 -> 92.40 $\pm$ 0.1 | No clear effect |
| Add the class token (ablation) | ProtoViT with greedy matching, without coherence loss or adjacency mask · CUB-200-2011 | accuracy: 84.32 $\pm$ 0.10 -> 85.54 $\pm$ 0.20 | Helped |
| Add the class token (ablation) | ProtoViT with coherence loss and adjacency mask · CUB-200-2011 | accuracy: 84.12 $\pm$ 0.05 -> 85.37 $\pm$ 0.13 | Helped |
| Add the coherence loss (ablation) | ProtoViT with class token and no adjacency mask · CUB-200-2011 | PLC: 35.88 $\pm$ 0.8 -> 32.98 $\pm$ 0.6 | Helped |
| Add the adjacency mask (ablation) | ProtoViT with class token and coherence loss · CUB-200-2011 | PLC: 32.98 $\pm$ 0.6 -> 21.68 $\pm$ 3.1 | Helped |
| Add the adjacency mask (ablation) | ProtoViT with class token and no coherence loss · CUB-200-2011 | accuracy: 85.45 $\pm$ 0.30 -> 85.37 $\pm$ 0.13 | Hurt |
| Add the coherence loss (ablation) | ProtoViT with class token and adjacency mask · CUB-200-2011 | accuracy: 85.45 $\pm$ 0.30 -> 85.37 $\pm$ 0.13 | Hurt |
Surprising findings
- Removing the adjacency mask slightly improves accuracy in the ablation, but damages prototype semantic coherence.
- Including the class token raises PLC (location shift) even though it improves accuracy.
Limitations
- Cannot provide explicit textual justification for prototypes.
- Not fully immune to location misalignment, similar to CNN-based prototype models.
- Prototype semantics may be hard to name in domains like radiology without natural textual descriptions.
Experimental setup
Datasets: CUB-200-2011 Stanford Cars
Benchmarks: CUB-200-2011 Stanford Cars Location Misalignment Benchmark
Metrics: accuracy PLC PRC PAC
Baselines: ProtoPNet Deformable ProtoPNet TesNet ProtoPool ProtoPFormer ViT-Net ProtoTree
Builds on: ProtoPNet TesNet Deformable ProtoPNet ProtoPFormer
Closest papers by meaning
- Interpretable Image Classification via Non-parametric Part Prototype Learning (CVPR 2025 · similarity 0.91)
- ProtoArgNet: Interpretable Image Classification with Super-Prototypes and Argumentation (AAAI 2025 · similarity 0.89)
- Improving Prototypical Visual Explanations with Reward Reweighing, Reselection, and Retraining (ICML 2024 · similarity 0.86)
- A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis (ICLR 2024 · similarity 0.85)
- Keep the Faith: Faithful Explanations in Convolutional Neural Networks for Case-Based Reasoning (AAAI 2024 · similarity 0.84)
- Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.