Research map / Architectures and efficiency / Attention and transformers
Transferring Visual Explainability from Self-Explaining to Prediction-Only Vision Transformers via Task Arithmetic
Yuya Yoshikawa, Ryotaro Shimizu, Takahiro Kawashima, Yuki Saito
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes transferring visual explainability from a self-explaining vision transformer to a prediction-only ViT via task arithmetic, avoiding training a new self-explaining model from scratch.
Problem
Users who already have prediction-only image classifiers face high labeling and computation costs to train a new self-explaining model from scratch when they also need efficient explanations.
Approach
Transfer the visual explanation capability into a prediction-only vision transformer using task arithmetic on model weights, rather than retraining a self-explaining model from scratch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add explainability vector from ImageNet+X to target-finetuned model | CLIP ViT-B/32 target model · 10 target datasets | E-RMSE / IoU@10: E-RMSE 1.562 -> 1.125; IoU@10 0.076 -> 0.279 | Improved |
| add explainability vector | target-finetuned model · 10 target datasets | accuracy: 0.890 -> 0.886 | No clear effect |
| add equal-norm random vector instead of explainability vector (ablation) | target model · Rank corr. | Rank corr.: -0.147 -> -0.144 | No clear effect |
| restrict vector to first six blocks (ablation) | explainability vector · Rank corr. | Rank corr.: -0.147 -> -0.147 | No clear effect |
| restrict vector to last six blocks (ablation) | explainability vector · IoU@1 | IoU@1: 0.009 -> 0.109 (full vector 0.132) | Improved |
| train vector with explanation maps shuffled across source images (ablation) | explainability vector · IoU@1 | IoU@1: 0.132 -> 0.079 | Hurt |
| increase explainability scaling lambda2 | transferred model · Cars accuracy | accuracy | Hurt |
| use learned attributions instead of Kernel SHAP-1000 | transferred model · IoU@10 vs GT | IoU@10: 0.233 -> 0.279 | Improved |
| regress transfer gain on six model-internal features (fixed source, held-out targets) | predictor · IoU@1 | held-out Pearson r: best r = +0.33 | No clear effect |
| regress transfer gain on model-internal features | predictor · E-MSE | held-out Pearson r: r = +0.92 (Lasso) | Improved |
Experimental setup
Builds on: task arithmetic
Closest papers by meaning
- Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models (ICML 2024 · similarity 0.88)
- Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention (AAAI 2024 · similarity 0.88)
- TVE: Learning Meta-attribution for Transferable Vision Explainer (ICML 2024 · similarity 0.88)
- On the Faithfulness of Vision Transformer Explanations (CVPR 2024 · similarity 0.87)
- Q-SENN: Quantized Self-Explaining Neural Networks (AAAI 2024 · similarity 0.86)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
