Research map / Architectures and efficiency / Attention and transformers

A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis

ICLR 2024 · Architectures and efficiency › Attention and transformers · approach: Efficient image transformers

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes INTR, a DETR-based Transformer encoder-decoder whose learned class-specific decoder queries let each class search for itself in an image, yielding faithful cross-attention interpretations for fine-grained classification.

Problem

Standard image classifiers incorporate class information only at the final fully-connected layer, so they cannot directly explain where the model looks to predict a particular class; post-hoc explanations are costly and low-resolution, and prior self-interpretable models need specialized architectur

Approach

Feed learnable class-specific queries (one per class) into a Transformer decoder so each class retrieves its own evidence via cross-attention, then score each class with a shared class-agnostic vector trained with cross-entropy; the cross-attention maps then serve as built-in, faithful interpretations, and multi-head attention captures class attributes.

Key results

• INTR reaches 97.4% on Bird vs ResNet-50 98.5%, 71.8% on CUB vs 83.8%, and 81.1% on Fish vs 71.9%, i.e., comparable on most datasets.
• On CUB, INTR insertion score 0.786 and deletion 0.01, beating Grad-CAM (0.61/0.31) and RISE (0.56/0.30) with a shared ResNet-50 classifier.
• In a human study, INTR attributes were identified with 74.7% accuracy versus 42.2% for ProtoPFormer.

Measured effects reported in the paper

Surprising findings

• Using ViT-Huge (ImageNet-21K) as encoder raises CUB accuracy from 71.8% to 83.2%, suggesting the drop is due to backbone and complex backgrounds rather than species recognition.
• Removing positional encoding from cross-attention keys was found necessary to avoid queries being restricted to particular grid locations and producing artifacts.

Limitations

• Accuracy drops notably on CUB (71.8% vs 83.8% for ResNet-50) due to complex backgrounds or small objects.
• Class-specific queries must be processed jointly, so training and inference cost may grow with a large number of classes C.
• Evaluation is mainly qualitative; the paper states high accuracy is not the goal.

Experimental setup

Datasets: Birds-525 CUB-200-2011 iNaturalist-2021 (Fish subset) Stanford Dogs Stanford Cars Oxford Pet FGVC-Aircraft Cambridge butterfly (Heliconiine collection) MSCOCO ImageNet-1K

Benchmarks: Birds-525 CUB-200-2011 Cambridge butterfly iNaturalist-2021 Fish Stanford Dogs Oxford Pet Stanford Cars FGVC-Aircraft

Metrics: classification accuracy insertion deletion attribute identification accuracy (human study)

Baselines: ResNet-50 Grad-CAM RISE ProtoPNet ProtoTree ProtoPFormer

Builds on: DETR Transformer ProtoPNet Grad-CAM RISE

Closest papers by meaning

• Analyzing Vision Transformers for Image Classification in Class Embedding Space (NeurIPS 2023 · similarity 0.89)
• TransFG: A Transformer Architecture for Fine-Grained Recognition (AAAI 2022 · similarity 0.89)
• ViT-NeT: Interpretable Vision Transformers with Neural Tree Decoder (ICML 2022 · similarity 0.86)
• Fine-Tuning Image Transformers Using Learnable Memory (CVPR 2022 · similarity 0.86)
• Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.86)
• Interpretable Image Classification with Adaptive Prototype-based Vision Transformers (NeurIPS 2024 · similarity 0.85)