Research map / Architectures and efficiency / Pruning and efficient tuning
Learning Where to Edit Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a meta-learned hypernetwork that predicts sparse binary masks over ViT feed-forward weights to locate where to edit, then fine-tunes the selected parameters to correct single predictive errors, evaluated on a new subpopulation-shift editing benchmark.
Problem
Correcting predictive errors of pre-trained vision transformers (ViTs) from subpopulation shifts while preserving generalization to neighboring failures and locality on unrelated inputs, without retraining.
Approach
A locate-then-edit method for ViTs where a hypernetwork, meta-trained on CutMix pseudo-samples with a KL reliability loss and an l1 sparsity penalty, outputs binary masks selecting structured rows/columns of FFN weights in layers 8-10; the how-to-edit step is then simple fine-tuning of those parameters with a threshold rho controlling the generalization-locality trade-off.
Key results
- Proposed method achieves the best generalization-locality Pareto front among compared editing methods on the ViT-B/16 editing benchmark.
- Edited layers 8-10 FFNs identified by greedy search give the best trade-off; MSA editing hurts locality.
- Multiple editing samples per group improve editing performance through mask ensembling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the learned hypernetwork mask to localize parameters for editing | ViT/B-16 model editing · Proposed editing benchmark | generalization-locality trade-off | Helped |
| Edit the 8th–10th FFNs rather than MSAs (ablation) | ViT/B-16 · 100 volleyball-to-basketball failures | generalization-locality trade-off | Helped |
| Increase parameter sparsity using the mask threshold (ablation) | KN, SPT, and the proposed method · Proposed editing benchmark | locality rate | Helped |
| Increase parameter sparsity using the mask threshold (ablation) | KN, SPT, and the proposed method · Proposed editing benchmark | generalization rate | Hurt |
| Replace the learned mask with a random mask at the same sparsity (ablation) | ViT/B-16 model editing · Proposed editing benchmark | editing performance | Hurt |
| Average masks from more editing samples in the same group (ablation) | Proposed method · Editing samples from the same group | editing performance | Helped |
| Generate pseudo-samples with CutMix instead of PGD (ablation) | Hypernetwork meta-training · Proposed editing benchmark, including AI-generated subsets | editing performance | No clear effect |
| Vary the outer-loop sparsity trade-off parameter (ablation) | Proposed method · Proposed editing benchmark | editing performance | No clear effect |
| Use one inner-loop gradient step instead of more steps (ablation) | Proposed method · ViT/B-16 editing | editing performance | Hurt |
| Increase inner-loop gradient steps from five to ten (ablation) | Proposed method · ViT/B-16 editing | editing performance | No clear effect |
| Use an inner-loop learning rate of 10^-4 rather than a larger rate (ablation) | Proposed method · ViT/B-16 editing | editing performance | Hurt |
| Reduce hypernetwork attention blocks from five to three or one (ablation) | Proposed method · ViT/B-16 editing | editing performance | No clear effect |
Surprising findings
- Simple CutMix pseudo-samples rival PGD adversarial pseudo-samples in simulating distribution shifts, including on AI-generated images.
- A hypernetwork with one to three attention blocks performs comparably to five-block version.
Limitations
- Only single-example editing is studied; batch editing is left for future work.
- CutMix pseudo-sample effectiveness is confirmed only empirically, without theoretical explanation.
- Evaluated mainly on ViT-B/16 and ViT-S/16 with object recognition; not tested on CNNs, Swin, or dense tasks.
Experimental setup
Datasets: ImageNet-1k ImageNet-21k ImageNet-R ImageNet-Sketch Flickr images (MAD-searched)
Benchmarks: ViT editing benchmark (18 groups: 16 MAD-searched natural, 2 AI-generated) ImageNet-1k/ImageNet-R/ImageNet-Sketch boundary samples for locality
Metrics: success rate generalization rate locality rate GR-LR Pareto curve
Baselines: FT FT-l2 T-Patcher KN SPT ROME LoRA KE MEND
Builds on: MEND ROME MALMEN CutMix MAML
Closest papers by meaning
- VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark (NeurIPS 2024 · similarity 0.88)
- Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit (AAAI 2025 · similarity 0.87)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.86)
- Split Adaptation for Pre-trained Vision Transformers (CVPR 2025 · similarity 0.86)
- ModHiFi: Identifying High Fidelity predictive components for Model Modification (NeurIPS 2025 · similarity 0.86)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.