Research map / Architectures and efficiency / CNN design
Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sub-explanation counting and cross-testing, built on search-based minimal sufficient explanations and insertion-based attribution metrics, to compare decision mechanisms of CNNs, ConvNeXt, and transformers on ImageNet, finding normalization choice strongly drives compositionality.
Problem
Understanding and comparing the decision-making mechanisms of CNNs and transformers at a dataset-wide level, beyond anecdotal per-image explanations.
Approach
Systematically applies beam-search minimal sufficient explanations and their sub-explanation trees over a dataset to count compositional vs disjunctive behavior, and cross-tests attribution maps between networks with insertion/deletion scores to build a feature-use similarity landscape.
Key results
- Batch normalization variant ConvNeXt-T-3-BN has about 80% fewer subexplanations at >=50% threshold than ConvNeXt-T-3-GN and ConvNeXt-T-3 (672.92 vs 3089.83 for ConvNeXt-T-3 and 7476.92 for ConvNeXt-T-3-GN).
- ConvNeXt-T has 5360.43 subexplanations at >=50% likelihood ratio versus 332.22 for ResNet50-D and 467.72 for DeiT-S-distilled.
- Averaging predictions of ConvNeXt-T, Swin-T and ResNet50-c2 gives 82.9% ImageNet accuracy, above each individual model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3×3 kernels for all ConvNeXt blocks (ablation) | ConvNeXt-T · ImageNet validation (first 5,000 images) | subexplanations ≥50%: 5360.43 -> 3089.83 | Hurt |
| Use 3×3 kernels and group normalization (ablation) | ConvNeXt-T · ImageNet validation (first 5,000 images) | subexplanations ≥50%: 5360.43 -> 7476.92 | Helped |
| Use 3×3 kernels and batch normalization (ablation) | ConvNeXt-T · ImageNet validation (first 5,000 images) | subexplanations ≥50%: 5360.43 -> 672.92 | Hurt |
| Use 4×4 windows in the first two stages (ablation) | Swin-T · ImageNet validation (first 5,000 images) | subexplanations ≥50%: 7268.20 -> 4276.08 | Hurt |
| Use 4×4 windows in the first two stages and group normalization (ablation) | Swin-T · ImageNet validation (first 5,000 images) | subexplanations ≥50%: 7268.20 -> 7039.34 | No clear effect |
| Use 4×4 windows in the first two stages and batch normalization (ablation) | Swin-T · ImageNet validation (first 5,000 images) | subexplanations ≥50%: 7268.20 -> 943.29 | Hurt |
| Distill the transformer from a CNN teacher | Transformers on ImageNet validation (first 5,000 images) · ImageNet validation (first 5,000 images) | subexplanations ≥50%: DeiT-S 2408.30 -> DeiT-S-distilled 467.72; PiT-S 3862.10 -> PiT-S-distilled 334.45 | Hurt |
| Use newer data augmentation | ResNet CNNs · ImageNet validation (first 5,000 images) | MSE count: 6.76 -> 9.52; 6.76 -> 11.01; 6.76 -> 9.88 | Helped |
| Average predictions from ConvNeXt-T, Swin-T, and ResNet50-C2 | Ensemble of ConvNeXt-T, Swin-T, and ResNet50-C2 · ImageNet | accuracy: 82.1%, 81.2%, 80.0% -> 82.9% | Helped |
Surprising findings
- Switching layer normalization to batch normalization reduced subexplanations by about 80%, a larger effect than reducing receptive field size (about 40%).
- Some subsets of an MSE yield even higher confidence than the full image, suggesting an occlusion handling mechanism distinct from compositional networks.
Limitations
- Evaluation uses only the first 5,000 ImageNet validation images due to slow explanation runs.
- Trends are statistical; individual images may show different inference strategies.
- Explanations depend on a 7x7 patch grid and blurred baseline, with normalization effects not fully resolved.
Experimental setup
Datasets: ImageNet validation (first 5,000 images)
Benchmarks: ImageNet
Metrics: MSE count MSE size sub-explanation count at likelihood ratio thresholds insertion score deletion score top-1 accuracy
Baselines: ResNet50 VGG19 ConvNeXt-T Swin-T Nest-T DeiT-S PiT-S LeViT-256
Builds on: Structured Attention Graphs iGOS++ insertion and deletion metrics Score-CAM
Closest papers by meaning
- Explaining Convolutional Neural Networks through Attribution-Based Input Sampling and Block-Wise Feature Aggregation (AAAI 2021 · similarity 0.87)
- Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models (ICML 2024 · similarity 0.87)
- Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (CVPR 2019 · similarity 0.87)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.87)
- Explaining in Diffusion: Explaining a Classifier with Diffusion Semantics (CVPR 2025 · similarity 0.85)
- Keep the Faith: Faithful Explanations in Convolutional Neural Networks for Case-Based Reasoning (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.