Research map / Vision-language and multimodal / CLIP and zero-shot
ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ArGue, an attribute-guided soft prompt tuning method for CLIP that aligns learnable prompts with LLM-generated visual attributes (with attribute sampling and a negative prompt variant ArGue-N) to improve novel-class and out-of-distribution generalization.
Problem
Soft prompt tuning of vision-language models learns class-name-anchored prompts that can rely on spurious correlations (e.g., background), hurting generalization to novel classes and distribution shifts.
Approach
Replace direct class-name soft prompts with prompts that combine class names and LLM-generated primitive visual attributes, select attributes via clustering and CLIP image similarity, and add a negative prompt (e.g., "the background of a {class}") trained toward uniform predictions to suppress spurious correlations.
Key results
- ArGue-N averages 81.18 harmonic mean across 11 novel-class datasets, +1.70 over the previous best (LASP).
- ArGue-N beats CLIP on novel classes in 10 of 11 datasets, with gains of +4.55 on FGVCAircraft and +3.98 on EuroSAT over the previous best.
- On ImageNet OOD transfer, ArGue-N reaches 65.02 on ImageNetV2, 49.25 on ImageNet-Sketch, 51.47 on ImageNet-A and 76.96 on ImageNet-R, consistently above LASP and CoCoOp.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace class-name-only prompt tuning with generated visual-attribute guidance (ablation) | CoOp · Average (11 datasets) | H: 71.66 -> 79.30 | Helped |
| Add prompt regularization to attribute-guided prompting (ablation) | Attribute-guided prompt tuning · Average (11 datasets) | H: 79.30 -> 80.44 | Helped |
| Select representative attributes by clustering and image similarity (ablation) | Attribute-guided prompt tuning with regularization · Average (11 datasets) | H: 80.44 -> 80.78 | Helped |
| Add negative prompting (ablation) | ArGue · Average (11 datasets) | H: 80.78 -> 81.18 | Helped |
| Use ArGue-N instead of LASP | CLIP ViT-B/16 · Average novel-class prediction | H: +1.70 | Helped |
| Add negative prompting to ArGue | ImageNet adaptation · ImageNetV2 | accuracy: 64.57 -> 65.02 | Helped |
| Add negative prompting to ArGue | ImageNet adaptation · ImageNet-Sketch | accuracy: 48.92 -> 49.25 | Helped |
| Add negative prompting to ArGue | ImageNet adaptation · ImageNet-A | accuracy: 50.93 -> 51.47 | Helped |
| Add negative prompting to ArGue | ImageNet adaptation · ImageNet-R | accuracy: 76.56 -> 76.96 | Helped |
| Add negative prompting to ArGue (ablation) | ArGue · DTD | H: 73.31 -> 73.08 | Hurt |
| Add negative prompting to ArGue (ablation) | ArGue · OxfordPets | H: 96.64 -> 97.40 | Helped |
Surprising findings
- Attribute-guided prompting alone (without regularization or sampling) gives a 7.64% average improvement over the CoOp baseline on novel-class prediction.
- Reducing attributes to 20% of the pool via sampling improves accuracy while lowering compute.
Limitations
- Negative prompt is a single general attribute ("the background of a"), a weak assumption that gives little gain on texture datasets such as DTD (-0.23%).
- Relies on LLM-generated attributes whose quality is uncertain and needs sampling to filter.
- Evaluated only with a ViT-B/16 CLIP backbone and 16-shot setting in the reported experiments.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp LASP
Builds on: CoOp LASP CoCoOp CLIP
Closest papers by meaning
- Prompt Tuning In a Compact Attribute Space (AAAI 2025 · similarity 0.93)
- Tree of Attributes Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.90)
- Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition (ICLR 2025 · similarity 0.90)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models (ICLR 2024 · similarity 0.89)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.