Research map / Vision-language and multimodal / CLIP and zero-shot
TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TPR, a CLIP-based generalized zero-shot learning model that adds a dual latent/attribute space built from a static-plus-learnable attribute reservoir and a Pearson-correlation topology-preserving loss that keeps CLIP's pairwise class angles.
Problem
Generalized zero-shot learning with VLMs, where test images must be classified among both seen and unseen classes without knowing the seen/unseen split; finetuning CLIP on seen classes causes weak generalization to unseen classes.
Approach
Augment the shared latent space with an attribute space induced by an attribute reservoir (static BERT-encoded attribute vocabulary plus learnable tokens), and regularize finetuning with a Pearson-based loss that preserves CLIP's pairwise class-angle topology across seen and unseen classes.
Key results
- TPR reaches H of 81.63 on AwA2, 32.53 on CUB, and 70.45 on FLO under GZSL, best H on 11 of 12 datasets.
- Compared with prompt learning methods, TPR gains on average 12.46 (S), 9.46 (U), and 11.94 (H) points in the GZSL setting.
- With EVA-02 backbone TPR reaches H of 86.39 on AwA2 and 40.36 on CUB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add attribute-space contrastive loss to latent-space contrastive training (ablation) | TPR · AwA2 | unseen accuracy: 68.14 -> 72.40 | Helped |
| Add attribute-space contrastive loss to latent-space contrastive training (ablation) | TPR · CUB | unseen accuracy: 23.24 -> 26.02 | Helped |
| Add attribute-space contrastive loss to latent-space contrastive training (ablation) | TPR · FLO | unseen accuracy: 60.45 -> 64.26 | Helped |
| Add topology-preserving loss to latent-space contrastive training (ablation) | TPR · AwA2 | unseen accuracy: 68.14 -> 74.50 | Helped |
| Add topology-preserving loss to latent-space contrastive training (ablation) | TPR · CUB | unseen accuracy: 23.24 -> 26.15 | Helped |
| Add topology-preserving loss to latent-space contrastive training (ablation) | TPR · FLO | unseen accuracy: 60.45 -> 62.26 | Helped |
| Combine static vocabulary with learnable attribute tokens instead of using only static vocabulary (ablation) | TPR · AwA2 | H: 80.56 -> 81.63 | Helped |
| Combine static vocabulary with learnable attribute tokens instead of using only learnable tokens (ablation) | TPR · AwA2 | H: 74.38 -> 81.63 | Helped |
| Replace topology-preserving loss with nuclear norm (ablation) | TPR · AwA2 | H: 81.63 -> 79.44 | Hurt |
| Replace topology-preserving loss with orthogonality constraint (ablation) | TPR · AwA2 | H: 81.63 -> 80.55 | Hurt |
| Apply topology-preserving loss in latent space instead of attribute space (ablation) | TPR · CUB | H: 32.53 -> 31.70 | Hurt |
| Increase the static attribute vocabulary size (ablation) | TPR · AwA2 | H | Helped |
Surprising findings
- Nuclear-norm maximization and orthogonality regularizers do not consistently improve performance, while the topology-preserving loss on the attribute space does.
- Removing most of the static attribute vocabulary barely hurts performance, since many vocabulary items are highly correlated.
Limitations
- Text-only descriptions may not capture fine-grained nuances (e.g., CUB), while attribute annotations are costly.
- Performance drops when descriptions are noisy or insufficiently specific, especially for generic scenes.
- The static attribute vocabulary is redundant and not condensed; the authors defer compact vocabulary construction to future work.
Experimental setup
Datasets: AwA2 CUB FLO SUN FGVC-Aircraft Country-211 StanfordCars EuroSAT DTD UCF101 Food101 OxfordPets
Benchmarks: AwA2 CUB FLO SUN FGVC-Aircraft Country-211 StanfordCars EuroSAT DTD UCF101 Food101 OxfordPets
Metrics: seen accuracy (S) unseen accuracy (U) harmonic mean (H)
Baselines: CLIP CoOp CoCoOp MaPLe PromptSRC ProGrad CE LSA ZLAP
Builds on: CLIP CoOp CoCoOp MaPLe PromptSRC
Closest papers by meaning
- Data-Free Generalized Zero-Shot Learning (AAAI 2024 · similarity 0.90)
- Improving Generalized Zero-Shot Learning by Exploring the Diverse Semantics from External Class Names (CVPR 2024 · similarity 0.90)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.90)
- S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment (AAAI 2024 · similarity 0.89)
- Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning (CVPR 2025 · similarity 0.89)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.