Research map / Label-efficient and robust learning / Recognition and long-tail
Towards Fine-Grained Interpretability: Counterfactual Explanations for Misclassification with Saliency Partition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FG-VCE, a non-generative fine-grained counterfactual explanation framework for misclassified images that uses a Saliency Partition module (Shapley-style approximation with localized Gaussian kernels) and candidate feature replacement from correctly classified samples.
Problem
Attribution maps for fine-grained classifiers are too coarse to explain why a model misclassifies a sample, since correct and misclassified predictions often receive indistinguishable highlights.
Approach
Approximate per-feature-point Shapley contributions by suppressing local regions with reversed Gaussian kernels (Saliency Partition), then iteratively replace the most contributive target features in the misclassified sample with the most semantically similar features from correctly classified samples until the prediction flips, and read invariant and dominant regions from the feature differences.
Key results
- On CUB-200-2011 with ResNet-50, FG-VCE reaches insertion 43.02 and deletion 10.27 (lower is better) and compact activation score 9.53, the best among compared methods.
- FG-VCE takes 5.2 s and 3.2 GB GPU memory per explanation versus 13.5 s and 5.4 GB for CCE on a single RTX A5000.
- Adding the Saliency Partition module to CCE improves its compact activation score from 7.85 to 9.16 on CUB with ResNet-50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FG-VCE instead of the strongest reported baseline on CUB-200-2011 with ResNet-50 | Fine-grained saliency explanation · CUB-200-2011, ResNet-50 | Ins.: 41.31 -> 43.02 | Helped |
| Use FG-VCE instead of the strongest reported baseline on CUB-200-2011 with ResNet-50 | Fine-grained saliency explanation · CUB-200-2011, ResNet-50 | Del.: 11.14 -> 10.27 | Helped |
| Use FG-VCE instead of the strongest reported baseline on CUB-200-2011 with ResNet-50 | Fine-grained saliency explanation · CUB-200-2011, ResNet-50 | ξ: 7.85 -> 9.53 | Helped |
| Use FG-VCE instead of the strongest reported baseline on CUB-200-2011 with VGG-16 | Fine-grained saliency explanation · CUB-200-2011, VGG-16 | Ins.: 39.26 -> 34.25 | Hurt |
| Use FG-VCE instead of the strongest reported baseline on CUB-200-2011 with VGG-16 | Fine-grained saliency explanation · CUB-200-2011, VGG-16 | Del.: 10.79 -> 9.89 | Helped |
| Use FG-VCE instead of the strongest reported baseline on CUB-200-2011 with VGG-16 | Fine-grained saliency explanation · CUB-200-2011, VGG-16 | ξ: 6.45 -> 8.96 | Helped |
| Use FG-VCE instead of the strongest reported baseline on Stanford Dogs with ResNet-50 | Fine-grained saliency explanation · Stanford Dogs, ResNet-50 | Ins.: 39.62 -> 42.15 | Helped |
| Use FG-VCE instead of the strongest reported baseline on Stanford Dogs with ResNet-50 | Fine-grained saliency explanation · Stanford Dogs, ResNet-50 | Del.: 11.24 -> 12.08 | Hurt |
| Use FG-VCE instead of the strongest reported baseline on Stanford Dogs with ResNet-50 | Fine-grained saliency explanation · Stanford Dogs, ResNet-50 | ξ: 9.12 -> 10.45 | Helped |
| Use FG-VCE instead of the strongest reported baseline on Stanford Dogs with VGG-16 | Fine-grained saliency explanation · Stanford Dogs, VGG-16 | Ins.: 39.02 -> 41.58 | Helped |
| Use FG-VCE instead of the strongest reported baseline on Stanford Dogs with VGG-16 | Fine-grained saliency explanation · Stanford Dogs, VGG-16 | Del.: 12.31 -> 12.06 | Helped |
| Add Saliency Partition (SP) to FG-VCE (ablation) | FG-VCE · CUB-200-2011, ResNet-50 | Ins.: 41.71 -> 43.02 | Helped |
Surprising findings
- The Saliency Partition module increases overall explanation generation time even though Top-m selection reduces it.
Limitations
- Evaluated only on CNN backbones (ResNet-50, VGG-16); transformer adaptability left for future work.
- Adding the SP module increases explanation generation time.
- Compact activation score is a new self-defined metric, and evaluation is on only two fine-grained datasets.
Experimental setup
Datasets: CUB-200-2011 Stanford Dogs
Benchmarks: CUB-200-2011 Stanford Dogs
Metrics: insertion deletion compact activation score explanation time GPU memory
Baselines: GBP IG RISE EP GradCAM GradCAM++ ScoreCAM FG-CAM CALM CE SCOUT CCE
Builds on: contrastive counterfactual explanation (CCE) SCOUT FG-CAM Grad-CAM
Closest papers by meaning
- Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differ (Abstract Reprint) (AAAI 2024 · similarity 0.88)
- Rethinking Visual Counterfactual Explanations Through Region Constraint (ICLR 2025 · similarity 0.88)
- V-CECE: Visual Counterfactual Explanations via Conceptual Edits (NeurIPS 2025 · similarity 0.88)
- Structure Your Data: Towards Semantic Graph Counterfactuals (ICML 2024 · similarity 0.88)
- Finer-CAM: Spotting the Difference Reveals Finer Details for Visual Explanation (CVPR 2025 · similarity 0.86)
- Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.