Research map / Vision-language and multimodal / CLIP and zero-shot
AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AWT, a training-free augmentation-weighting-optimal-transport framework that enriches image views and LLM class descriptions, weights them by prediction entropy, and matches them via Sinkhorn OT to improve zero-shot and few-shot transfer of CLIP-style VLMs.
Problem
Pre-trained VLMs such as CLIP underperform on new classes when adapted with only raw images and class names, and prior adaptation methods either need training or augment only one modality.
Approach
Augment both modalities (random crops/flips and two-step dataset-aware LLM descriptions), weight each view by negative prediction entropy, and compute image-text distance as an entropic optimal transport problem with cosine cost, with a multimodal adapter for few-shot learning.
Key results
- Zero-shot image classification averages 69.59% over 14 datasets, versus 63.06% for CLIP and 67.54% for the best prior method SuS-X-SD.
- OOD generalization average of 64.43% across four ImageNet variants, above TPT at 60.81%.
- Zero-shot video: 56.1% on HMDB51 EP1 and 76.1% on Kinetics-600, exceeding FROSTER on these splits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace raw inputs with AWT augmentation, entropy weighting, and optimal transport | CLIP-B/16 · Zero-shot image classification (Avg. 14) | top-1 accuracy (%): 63.06 -> 69.59 | Helped |
| Use AWT instead of the strongest listed baseline | CLIP-B/16 · OOD generalization (OOD) | top-1 accuracy (%): 60.81 -> 64.43 | Helped |
| Add class-name description augmentation (ablation) | AWT on CLIP-B/16 · Zero-shot image classification (Avg. 14) | top-1 accuracy (%): 63.06 -> 67.36 | Helped |
| Add image augmentation only (ablation) | AWT on CLIP-B/16 · Zero-shot image classification (Avg. 14) | top-1 accuracy (%): 63.06 -> 63.40 | Helped |
| Add image augmentation only (ablation) | AWT on CLIP-B/16 · OOD generalization (OOD) | top-1 accuracy (%): 57.15 -> 56.77 | Hurt |
| Add optimal transport to the augmented inputs (ablation) | AWT on CLIP-B/16 · OOD generalization (OOD) | top-1 accuracy (%): 59.92 -> 60.48 | Helped |
| Add entropy-based view weighting to augmentation and optimal transport (ablation) | AWT on CLIP-B/16 · OOD generalization (OOD) | top-1 accuracy (%): 60.48 -> 64.43 | Helped |
| Use 100 augmented image views instead of 50 (ablation) | AWT on CLIP-B/16 · OOD generalization (OOD) | top-1 accuracy (%): 64.43 -> 64.54 | Helped |
| Use 100 augmented image views instead of 50 (ablation) | AWT on CLIP-B/16 · Zero-shot image classification (Avg. 14) | top-1 accuracy (%): 69.59 -> 69.52 | Hurt |
| Use the dataset-aware two-step LLM prompt instead of VisDesc prompting (ablation) | AWT on CLIP-B/16 · Zero-shot image classification (Avg. 14) | top-1 accuracy (%): 67.78 -> 69.59 | Helped |
| Lower visual weighting temperature from 100 to 1/2 (ablation) | AWT on CLIP-B/16 · OOD generalization (OOD) | top-1 accuracy (%): 61.07 -> 64.43 | Helped |
| Use a multi-modal adapter instead of a visual-only adapter (ablation) | AWT few-shot ImageNet · ImageNet (1-shot) | top-1 accuracy (%): 71.66 -> 72.14 | Helped |
Surprising findings
- Directly augmenting images with random crops and flips alone did not improve over raw CLIP (63.40 vs 63.06 average) and hurt OOD performance, attributed to background crops.
- On low-resolution CIFAR-10/100, random resized crop degraded AWT (-0.91 and -0.17), and replacing it with DALL·E 2 augmentation was needed to gain.
Limitations
- Inference cost increases with augmented views, reducing speed.
- Random resized cropping blurs low-resolution images such as CIFAR, hurting performance unless a diffusion model is used for augmentation.
- Few-shot adapter setup is slower than prompt-learning baselines due to sequential adapter processing.
Experimental setup
Datasets: ImageNet Caltech101 Caltech256 OxfordPets StanfordCars OxfordFlowers Food101 FGVCAircraft Birdsnap CUB SUN397 DTD
Benchmarks: ImageNet Caltech101 Caltech256 OxfordPets StanfordCars OxfordFlowers Food101 FGVCAircraft Birdsnap CUB SUN397 DTD
Metrics: top-1 accuracy average accuracy standard deviation FPS
Baselines: CLIP CoOp CoCoOp MaPLe PLOT++ POMP ProVP-Ref TPT DiffTPT PromptAlign Self-TPT CuPL
Builds on: CLIP CoOp TPT PLOT Open-VCLIP Sinkhorn algorithm
Closest papers by meaning
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.90)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.90)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.89)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.