Research map / Vision-language and multimodal / CLIP and zero-shot
Attend and Enrich: Enhanced Visual Prompt for Zero-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AENet, a prompt-learning zero-shot learning method that distills a semantic-enhanced visual prompt via concept-aware attention and a linear residual refinement unit with attribute consistency loss.
Problem
Prompt-based zero-shot learning over-emphasizes primary visual features learned on seen classes, which limits generalization to unseen classes and leaves semantic-related visual details under-used.
Approach
Harmonize visual and attribute tokens through a shared learnable modal token (concept-aware attention), then predict residual semantic details with a zero-initialized linear unit (VRRU) supervised by attribute consistency, and add these residuals to the visual prompt before combining with visual tokens.
Key results
- AENet reaches 80.3% ZSL accuracy on CUB, 70.4% on SUN and 75.2% on AwA2, reported as state-of-the-art.
- GZSL harmonic mean H of 74.7% (CUB), 51.0% (SUN) and 74.9% (AwA2).
- Removing the learnable prompt P drops CUB ZSL accuracy from 80.3% to 70.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learnable prompt P (ablation) | AENet · CUB, SUN, AwA2 | ZSL accuracy: 70.2 -> 80.3; 64.5 -> 70.4; 70.7 -> 75.2 | Helped |
| Add learnable prompt P (ablation) | AENet · CUB, SUN, AwA2 | GZSL H: 68.0 -> 74.7; 34.8 -> 51.0; 72.1 -> 74.9 | Helped |
| Add residual details Z (ablation) | AENet · CUB, SUN, AwA2 | ZSL accuracy: 75.8 -> 80.3; 68.0 -> 70.4; 73.2 -> 75.2 | Helped |
| Add residual details Z (ablation) | AENet · CUB, SUN, AwA2 | GZSL H: 72.5 -> 74.7; 46.2 -> 51.0; 74.0 -> 74.9 | Helped |
| Add concept-aware attention (ablation) | AENet · CUB, SUN, AwA2 | ZSL accuracy: 78.8 -> 80.3; 68.8 -> 70.4; 73.7 -> 75.2 | Helped |
| Add concept-aware attention (ablation) | AENet · CUB, SUN, AwA2 | GZSL H: 73.8 -> 74.7; 48.6 -> 51.0; 73.5 -> 74.9 | Helped |
| Replace ZLinear with a linear layer (ablation) | AENet prompt distillation · CUB, AwA2 | ZSL accuracy: 79.8 vs 80.3; 74.1 vs 75.2 | Hurt |
| Replace ZLinear with MLP (ablation) | AENet prompt distillation · CUB, AwA2 | ZSL accuracy: 79.5 vs 80.3; 74.8 vs 75.2 | Hurt |
| Replace skip connection with gated skip (ablation) | AENet prompt distillation · CUB, AwA2 | ZSL accuracy: 79.5 vs 80.3; 73.5 vs 75.2 | Hurt |
| Set prompt length T to approximately 5 instead of other tested lengths (ablation) | AENet · CUB, SUN, AwA2 | accuracy and H | Helped |
| Add attribute consistency loss with λ_cons rising from 0.0 to 1.0 (ablation) | AENet · CUB, SUN, AwA2 | GZSL H | Helped |
| Increase λ_cons above 1.0 (ablation) | AENet · CUB, SUN, AwA2 | GZSL H | Hurt |
Surprising findings
- More complex residual-unit variants (MLP, gated skip) did not improve over the simple zero-initialized linear unit.
Limitations
- Evaluated only on three small attribute-based benchmarks (CUB, SUN, AwA2).
- Relies on per-class attribute annotations, so it is not applicable where attributes are unavailable.
Experimental setup
Datasets: CUB SUN AwA2
Benchmarks: CUB SUN AwA2
Metrics: top-1 accuracy (ZSL) harmonic mean H (GZSL) unseen accuracy U seen accuracy S
Baselines: ZSLViT TransZero++ DUET I2MVFormer DPPN GEM-ZSL f-VAEGAN+DSP CoOp
Builds on: ViT GloVe LLaVA calibrated stacking VPT
Closest papers by meaning
- Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning (CVPR 2025 · similarity 0.92)
- Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning (CVPR 2024 · similarity 0.91)
- ZeroDiff: Solidified Visual-semantic Correlation in Zero-Shot Learning (ICLR 2025 · similarity 0.88)
- ZeroMamba: Exploring Visual State Space Model for Zero-Shot Learning (AAAI 2025 · similarity 0.88)
- Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning (CVPR 2024 · similarity 0.88)
- Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.