Research map / Vision-language and multimodal / CLIP and zero-shot

Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VSPCN, a ViT-based GZSL framework that adapts a pre-trained ViT-Base with learnable visual and semantic prompts, fused via weak (shallow) and strong (deep) fusion mechanisms, and reports state-of-the-art results on CUB, SUN and AWA2.

Problem

Generalized zero-shot image recognition suffers from overfitting seen classes when the visual backbone is fine-tuned on limited seen-class data, and visual-semantic alignment is weak in intermediate layers.

Approach

Replaces backbone fine-tuning with prompt tuning: a visual prompt that gathers discriminative image information and a semantic prompt that gathers attribute information are injected into ViT layers via weak fusion in shallow layers and strong attention-with-bias fusion in deeper layers, with an adapter updating attributes per instance.

Key results

• Achieves GZSL harmonic mean H of 75.7% on CUB, 53.8% on SUN and 77.6% on AWA2, outperforming PSVMA by 2.9, 1.5 and 2.2 points.
• Achieves CZSL accuracy of 80.6% on CUB, 75.3% on SUN and 76.6% on AWA2, the best among compared methods.
• Ablation shows the semantic prompt gives larger H gains than the visual prompt alone (e.g., +14.6 vs +13.4 on CUB), and removing the semantic adapter lowers performance.

Measured effects reported in the paper

Surprising findings

• Semantic prompt alone yields larger gains on AWA2 (+11.2 H) than visual prompt alone (+3.6 H).

Limitations

• Relies on GloVe attribute embeddings and class attribute annotations, limiting applicability to settings without attributes.
• Many hyperparameters (lambda_CED, lambda_SKD, alpha_v, alpha_s, tau) tuned on benchmark splits.
• Evaluated only on three standard attribute-based GZSL datasets with ViT-Base pretrained on ImageNet-1k.

Experimental setup

Datasets: CUB SUN AWA2

Benchmarks: CUB SUN AWA2

Metrics: accuracy (CZSL) seen accuracy S unseen accuracy U harmonic mean H

Baselines: LsrGAN CE-GZSL FREE HSVA ICCE VS-Boost DAZLE APN GEM-ZSL DPPN TransZero MSDN

Builds on: PSVMA ZSLViT SP (semantic prompt for few-shot) DFSP

Closest papers by meaning

• Attend and Enrich: Enhanced Visual Prompt for Zero-Shot Learning (AAAI 2025 · similarity 0.92)
• Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning (CVPR 2024 · similarity 0.90)
• ZeroMamba: Exploring Visual State Space Model for Zero-Shot Learning (AAAI 2025 · similarity 0.90)
• TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.89)
• Improving Generalized Zero-Shot Learning by Exploring the Diverse Semantics from External Class Names (CVPR 2024 · similarity 0.89)
• Class Semantic Attribute Perception Guided Zero-Shot Learning (AAAI 2025 · similarity 0.89)