Research map / Vision-language and multimodal / CLIP and zero-shot
Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZSLViT, a ViT backbone for zero-shot learning that inserts semantic-embedded token learning and visual enhancement into encoder layers to progressively keep semantic-related visual tokens and fuse semantic-unrelated ones.
Problem
Zero-shot learning with pre-trained CNN or ViT backbones learns visual features not matched to semantic attributes, leading to poor visual-semantic interactions and transfer to unseen classes.
Approach
Makes the ViT encoder semantic-aware: visual-semantic consistency learning (MLPs with reconstruction losses) and semantic embedding enhance the [cls] token, whose attention to other tokens identifies semantic-related tokens to keep, while low-correspondence tokens are fused into one token at multiple layers.
Key results
- Achieves 78.9% CZSL accuracy on CUB and 73.6% GZSL harmonic mean on CUB, the best among compared methods.
- Reports 47.3% GZSL harmonic mean on SUN and 74.2% on AWA2, the best results in Table 1.
- Ablation on CUB shows removing the visual reconstruction loss L_VR drops harmonic mean from 73.6 to 63.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZSLViT instead of the strongest reported baseline | ZSLViT · CUB | acc: 77.8 -> 78.9 | Helped |
| Use ZSLViT instead of the strongest reported baseline | ZSLViT · SUN | acc: 66.0 -> 68.3 | Helped |
| Use ZSLViT instead of the strongest reported baseline | ZSLViT · AWA2 | H: 73.8 -> 74.2 | Helped |
| Add visual reconstruction loss LV R (ablation) | ZSLViT · CUB | acc: 76.4 -> 78.9 | Helped |
| Add visual reconstruction loss LV R (ablation) | ZSLViT · AWA2 | acc: 61.4 -> 70.7 | Helped |
| Add semantic reconstruction loss LSR (ablation) | ZSLViT · CUB | acc: 78.0 -> 78.9 | Helped |
| Add semantic reconstruction loss LSR (ablation) | ZSLViT · AWA2 | acc: 70.2 -> 70.7 | Helped |
| Add semantic embedding (ablation) | ZSLViT · CUB | acc: 76.2 -> 78.9 | Helped |
| Add semantic embedding (ablation) | ZSLViT · AWA2 | acc: 69.9 -> 70.7 | Helped |
| Add visual enhancement (ViE) (ablation) | ZSLViT · CUB | acc: 77.9 -> 78.9 | Helped |
| Add visual enhancement (ViE) (ablation) | ZSLViT · AWA2 | acc: 70.5 -> 70.7 | Helped |
Surprising findings
- Gains on the coarse-grained AWA2 dataset in CZSL are not comparable to the fine-grained gains, attributed to visual-unrelated attributes such as 'eat fish'.
Limitations
- Gains on AWA2 are smaller because many attributes are visually unrelated to images.
- Relies on a fixed fusing rate kappa and layer selection (4th, 7th, 10th encoders) chosen by default settings.
Experimental setup
Datasets: CUB SUN AWA2
Benchmarks: CUB SUN AWA2
Metrics: top-1 accuracy (CZSL) harmonic mean H (GZSL) seen accuracy S unseen accuracy U
Baselines: AREN f-VAEGAN TF-VAEGAN APN MSDN TransZero CLIP CoOp I2DFormer I2MVFormer DUET GEM-ZSL
Builds on: Vision Transformer (ViT) APN attribute-based cross-entropy loss I2MVFormer DUET
Closest papers by meaning
- ZeroMamba: Exploring Visual State Space Model for Zero-Shot Learning (AAAI 2025 · similarity 0.93)
- Attend and Enrich: Enhanced Visual Prompt for Zero-Shot Learning (AAAI 2025 · similarity 0.91)
- ZeroDiff: Solidified Visual-semantic Correlation in Zero-Shot Learning (ICLR 2025 · similarity 0.91)
- Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning (CVPR 2025 · similarity 0.90)
- Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning (CVPR 2024 · similarity 0.90)
- Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.