Research map / Label-efficient and robust learning / Recognition and long-tail
TVE: Learning Meta-attribution for Transferable Vision Explainer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a transferable DNN explainer (TVE) on ImageNet to predict patch-level "meta-attribution" tensors from a frozen ViT-style backbone, then transfers to downstream vision classifiers via a rule using the task classifier head, without task-specific explainer training.
Problem
Existing DNN-based explainers for vision models are tied to a single model and task, so each new downstream task requires training a new explainer, which is time- and resource-intensive.
Approach
Define meta-attribution as backbone-encoder embeddings of neighborhood-masked and background-masked inputs; pre-train an explainer to predict these tensors on ImageNet, then explain any downstream task by taking log-differences through the task-specific classifier head, requiring no task-specific explainer training.
Key results
- On ViT-Base over CIFAR-10, TVE achieves Fidelity+ AUC 0.40 and Fidelity- AUC 0.31, versus 0.36/0.36 for ViTShapley.
- TVE outperforms ViTShapley on Fidelity+ and Fidelity- for most of the 9 architecture-dataset combinations, e.g. Swin-Base on CIFAR-10 0.31/0.24 vs 0.25/0.28.
- TVE explains with a single feed-forward pass and its throughput degrades least when moving from ViT-Base to ViT-Large, per the authors' latency study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the task-specific classifier for attribution transfer instead of the general ImageNet-pretrained classifier (ablation) | TVE · Cats-vs-dogs, Imagenette, and CIFAR-10 | Fidelity+ and Fidelity- sparsity AUC | Mixed |
| pre-train the transferable explainer on ImageNet rather than use a randomly initialized explainer (ablation) | TVE · Cats-vs-dogs, Imagenette, and CIFAR-10 | Fidelity | Helped |
| use the task-specific classifier instead of the ImageNet general classifier (ablation) | TVE · CIFAR-10; DeiT-Base | Fidelity+ sparsity AUC: 0.32$\pm$ -> 0.30$\pm$ | Hurt |
Surprising findings
- Explainer transfer without any downstream training reaches fidelity competitive with explainers trained on the task, and TVE-PT is competitive with TVE-FT on fully fine-tuned models.
Limitations
- Transfer is only argued to hold for downstream tasks within the pre-training data distribution.
- Approximation of the exact attribution by two antithetical states (full complement and empty set) is validated only empirically; the theoretical error bound depends on an assumed epsilon range.
- Evaluation covers only three ViT-style classifiers and three small downstream datasets.
Experimental setup
Datasets: ImageNet Cats-vs-dogs CIFAR-10 Imagenette
Benchmarks: Cats-vs-dogs Imagenette CIFAR-10 ImageNet
Metrics: Fidelity+ sparsity AUC Fidelity- sparsity AUC throughput
Baselines: ViT-Shapley LIME Integrated Gradients RISE DeepLift KernelSHAP GradShap
Builds on: ViT-Shapley CoRTX FastSHAP V-information
Closest papers by meaning
- Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models (ICML 2024 · similarity 0.85)
- ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts (ICLR 2025 · similarity 0.85)
- How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations (ICLR 2025 · similarity 0.84)
- Sparse autoencoders reveal selective remapping of visual concepts during adaptation (ICLR 2025 · similarity 0.84)
- Benchmarking the Attribution Quality of Vision Models (NeurIPS 2024 · similarity 0.84)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.