Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Patching open-vocabulary models by interpolating weights
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, Ludwig Schmidt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| interpolate weights of fine-tuned and zero-shot model (PAINT) instead of using the zero-shot model | CLIP ViT-L/14 · nine patching tasks | accuracy: +0.8 to +19.4 (Table 1 held-out classes); abstract: 15 to 60 points on tasks | Improved |
| patch on task A, measure disjoint-class task B | PAINT ViT-L/14 · EuroSAT/RESISC45 pair | accuracy: 71.0 -> 69.7 | Hurt |
| patch on task A, measure related task B | PAINT ViT-L/14 · seven other task pairs | accuracy: +1.1 to +19.2 | Improved |
| scale up model size (ablation) | PAINT · patching effectiveness | accuracy distance to optimal | Improved |
| use joint patching versus unpatched (ablation) | ViT-L/14 · nine patching tasks + ImageNet | combined accuracy: +15.8 points | Improved |
| joint patching versus specialized models (ablation) | ViT-L/14 · combined accuracy | combined accuracy: 0.5 points worse | No clear effect |
| add weight interpolation to sequential fine-tuning (ablation) | ViT-B/32 sequential patching · combined accuracy | combined accuracy: -4.6 -> +11 points vs unpatched | Improved |
| parallel patching instead of joint (ablation) | PAINT multi-task · nine tasks | combined accuracy | Hurt |
| black-box optimization of mixing coefficients versus shared coefficient (ablation) | parallel patching · nine tasks | accuracy: 0.3 to 0.4 points | No clear effect |
| use ResNets instead of ViTs (ablation) | PAINT · patching effectiveness | accuracy trade-off | Hurt |
| patch on synthetic typographic attacks | ViT-L/14 · real-world typographic attacks | accuracy: +41 points | Improved |
| patch on counting subset of classes | ViT-L/14 · CLEVR unseen counts | accuracy: 59% -> over 99% | Improved |
Closest papers by meaning
- Understanding Layer Patching in Model Size Interpolation (NeurIPS 2026 · similarity 0.87)
- Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection (CVPR 2025 · similarity 0.87)
- Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptions (NeurIPS 2025 · similarity 0.87)
- LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.86)
- Transferable and Principled Efficiency for Open-Vocabulary Segmentation (CVPR 2024 · similarity 0.86)
- A Simple Framework for Open-Vocabulary Zero-Shot Segmentation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
