AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Transferring Visual Explainability from Self-Explaining to Prediction-Only Vision Transformers via Task Arithmetic

Yuya Yoshikawa, Ryotaro Shimizu, Takahiro Kawashima, Yuki Saito

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes transferring visual explainability from a self-explaining vision transformer to a prediction-only ViT via task arithmetic, avoiding training a new self-explaining model from scratch.

Problem

Users who already have prediction-only image classifiers face high labeling and computation costs to train a new self-explaining model from scratch when they also need efficient explanations.

Approach

Transfer the visual explanation capability into a prediction-only vision transformer using task arithmetic on model weights, rather than retraining a self-explaining model from scratch.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add explainability vector from ImageNet+X to target-finetuned modelCLIP ViT-B/32 target model · 10 target datasetsE-RMSE / IoU@10: E-RMSE 1.562 -> 1.125; IoU@10 0.076 -> 0.279Improved
add explainability vectortarget-finetuned model · 10 target datasetsaccuracy: 0.890 -> 0.886No clear effect
add equal-norm random vector instead of explainability vector (ablation)target model · Rank corr.Rank corr.: -0.147 -> -0.144No clear effect
restrict vector to first six blocks (ablation)explainability vector · Rank corr.Rank corr.: -0.147 -> -0.147No clear effect
restrict vector to last six blocks (ablation)explainability vector · IoU@1IoU@1: 0.009 -> 0.109 (full vector 0.132)Improved
train vector with explanation maps shuffled across source images (ablation)explainability vector · IoU@1IoU@1: 0.132 -> 0.079Hurt
increase explainability scaling lambda2transferred model · Cars accuracyaccuracyHurt
use learned attributions instead of Kernel SHAP-1000transferred model · IoU@10 vs GTIoU@10: 0.233 -> 0.279Improved
regress transfer gain on six model-internal features (fixed source, held-out targets)predictor · IoU@1held-out Pearson r: best r = +0.33No clear effect
regress transfer gain on model-internal featurespredictor · E-MSEheld-out Pearson r: r = +0.92 (Lasso)Improved

Experimental setup

Builds on: task arithmetic

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy