Research map / Vision-language and multimodal / CLIP and zero-shot

Attribute-based Visual Reprogramming for Vision-Language Models

ICLR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AttrVR, which replaces template-prompted labels in CLIP visual reprogramming with LLM-generated descriptive and distinctive attributes, sample-specific k-nearest attribute selection re-queried each epoch, and reports gains over VP and AR on 12 classification datasets.

Problem

Existing visual reprogramming (VR) methods for CLIP reuse the unimodal-vision pipeline, supervising noise patterns only with template-prompted ground-truth labels, which have similar text embeddings and cause misclassification in few-shot settings.

Approach

Use LLM-generated descriptive attributes (common features) and distinctive attributes (unique features) per class as supervision instead of label templates, and iteratively select the k nearest attributes per image at each epoch so that image-attribute alignment adapts as the VR pattern evolves; theory argues this reduces intra-class variance and increases inter-class separation.

Key results

• With ViT-B/16 CLIP, 16-shot AttrVR averages 77.7% across 12 datasets vs 75.7% for AR and 73.5% for VP.
• AttrVR gives the best mean accuracy over backbones RN50 (64.2), RN101 (66.8), ViT-B/32 (69.1), ViT-B/16 (77.7) and ViT-L/14 (75.5), beating AR in each.
• Zero-shot AttrZS (no VR training) already beats few-shot VP and AR on some datasets such as Caltech, Food and SUN.

Measured effects reported in the paper

Surprising findings

• Label-based VP degrades with ViT-L/14 (61.1% average) below its zero-shot baseline (68.7%)
• Attribute-only zero-shot classification (AttrZS) outperforms trained label-based VR baselines on several datasets

Limitations

• Relies on LLM-generated attributes whose quality affects performance; the paper only partially addresses low-quality attributes in the appendix
• Gains are modest on some datasets (e.g., Food, Cars, where AttrVR is at or below AttrZS or AR), and the theory relies on assumptions such as attribute-embedding closeness to class means
• Attribute generation uses GPT-3.5, a closed API, and the attribute sets are fixed per class

Experimental setup

Datasets: FGVCAircraft Caltech101 StanfordCars DTD EuroSAT Flowers102 Food101 OxfordPets SUN397 UCF101 ImageNet Resisc45

Benchmarks: FGVCAircraft Caltech101 StanfordCars DTD EuroSAT Flowers102 Food101 OxfordPets SUN397 UCF101 ImageNet Resisc45

Metrics: accuracy

Baselines: ZS AttrZS VP AR

Builds on: CLIP visual reprogramming (AR) visual prompting (VP) LaBo CoOp

Closest papers by meaning

• Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts (ICML 2025 · similarity 0.90)
• Bayesian-guided Label Mapping for Visual Reprogramming (NeurIPS 2024 · similarity 0.88)
• Sample-specific Masks for Visual Reprogramming-based Prompting (ICML 2024 · similarity 0.86)
• ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts (ICLR 2025 · similarity 0.86)
• UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models (NeurIPS 2024 · similarity 0.86)
• Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.85)