Research map / Label-efficient and robust learning / Self-supervised and contrastive
How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the training loss (BCE vs CE) and complexity (B-cos MLP) of a frozen-backbone linear/MLP probe strongly change post-hoc explanation quality, more than the pre-training scheme, and proposes BCE-trained B-cos MLP probes to improve explanations.
Problem
Post-hoc attribution methods are assumed to be applicable independently of how a model was trained, but explanation quality may depend strongly on how the classification probe on a pre-trained backbone is trained.
Approach
Explanations of frozen-backbone probes depend heavily on the probe's objective: CE is shift-invariant so explanations are poorly class-specific, while BCE is not shift-invariant and yields better-localized attributions; interpretable B-cos MLP probes further improve localization and accuracy.
Key results
- For LRP on DINO conventional ResNet-50, GridPG rises from 52% (CE) to 92% (BCE) with a linear probe.
- For IntGrad with DINO, moving from a linear probe to a 3-layer B-cos MLP raises GridPG from 70.0 to 89.0 while accuracy goes from 70.6 to 73.1.
- For ViT-B/16 DINO with CGW1, BCE probes improve GridPG by up to 12 p.p. over CE probes on conventional ViTs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the linear probe with BCE instead of CE (ablation) | ResNet-50 DINO with LRP explanations · ImageNet | GridPG: 52% -> 92% | Helped |
| Train the linear probe with BCE instead of CE (ablation) | ResNet-50 CLIP with LRP explanations · ImageNet | GridPG: 19%→51% | Helped |
| Train the linear probe with BCE instead of CE (ablation) | ViT-B/16 with CGW1 explanations · ImageNet | GridPG: 31%→43% | Helped |
| Train the linear probe with BCE instead of CE (ablation) | ViT-B/16 with Rollout explanations · ImageNet | EPG, bounding box size <50%: 53.3 -> 52.6; (-0.7) | Hurt |
| Train the linear probe with BCE instead of CE (ablation) | ViT-B/16 with Rollout explanations · ImageNet | EPG, bounding box size <75%: 61.1 -> 60.4; (-0.7) | Hurt |
| Train the linear probe with BCE instead of CE (ablation) | ViT-B/16 with CausalX explanations · ImageNet | EPG: 14.6 -> 14.6 (0.0); 28.6 -> 29.1 (+0.5); 42.2 -> 41.6 (-0.6); 61.2 -> 61.4 (+(0.2) | Mixed |
| Use a 3-layer B-cos MLP probe instead of a linear probe (ablation) | B-ViT_c-S/16 DINO · ImageNet | localization: +2.5 | Helped |
| Use a 3-layer B-cos MLP probe instead of a linear probe (ablation) | B-ViT_c-B/16 DINO · ImageNet | localization: +3.1 | Helped |
| Use conventional MLP probes instead of a linear probe (ablation) | Conventional backbones · ImageNet | GridPG | Hurt |
| Shift all class probe weights by a shared vector under CE (ablation) | Functionally equivalent linear probes with LRP · ImageNet | GridPG: 65.7% vs. 11.9% | Mixed |
Surprising findings
- Two functionally equivalent linear probes (same predictions and loss) yield GridPG scores of 65.7% vs 11.9%, showing CE-trained probe explanations are arbitrary due to shift invariance.
- Training loss of the probe impacts explanation quality more than the pre-training scheme itself.
Limitations
- Conventional MLP probes decrease GridPG on ImageNet, so benefits of non-linear probes are not universal; GradCAM and Input x Gradient improvements are inconsistent for conventional models.
- Evaluation relies on heuristic localization metrics (GridPG, EPG) and pixel deletion rather than human judgment.
Experimental setup
Datasets: ImageNet VOC COCO
Benchmarks: ImageNet VOC COCO
Metrics: GridPG EPG pixel deletion Gini index compactness entropy complexity top-1 accuracy
Baselines: Cross-entropy linear probe GradCAM Integrated Gradients LIME CausalX CGW1 Rollout
Builds on: B-cos networks Layer-wise Relevance Propagation Grid pointing game Energy pointing game
Closest papers by meaning
- Learning Deep Attribution Priors Based On Prior Knowledge (NeurIPS 2020 · similarity 0.90)
- Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretability (NeurIPS 2023 · similarity 0.89)
- Explaining Convolutional Neural Networks through Attribution-Based Input Sampling and Block-Wise Feature Aggregation (AAAI 2021 · similarity 0.89)
- Towards Better Understanding Attribution Methods (CVPR 2022 · similarity 0.89)
- Provably Better Explanations with Optimized Aggregation of Feature Attributions (ICML 2024 · similarity 0.88)
- Benchmarking the Attribution Quality of Vision Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.