Research map / Vision-language and multimodal / CLIP and zero-shot
TOMCAT: Test-time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TOMCAT, a test-time method that accumulates textual and visual prototype knowledge from unlabeled test images via knowledge accumulation modules, a priority queue, and adaptive update weights to handle label-space shift in compositional zero-shot learning.
Problem
Compositional zero-shot learning suffers performance degradation because the label space at test time includes unseen attribute-object compositions, causing distribution shift that frozen prototypes cannot adapt to.
Approach
Keep textual prototypes frozen as a base, learn lightweight knowledge accumulation modules that update both textual and visual (queue-based, high-confidence image) prototypes at test time, with an adaptive weight based on image-prototype similarity, trained with prediction entropy minimization and a multimodal contrastive alignment loss.
Key results
- Closed-world AUC on UT-Zappos 48.3 vs. 46.6 for second-best ClusPro; C-GQA AUC 16.0 vs. 14.9.
- Open-world AUC on UT-Zappos 43.7 vs. 39.5 for ClusPro; Clothing16K open-world AUC 95.8 vs. 91.8 for ATIF.
- On MIT-States, TOMCAT (AUC 22.6) falls behind ClusPro (23.8) in closed-world, attributed to label noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TOMCAT rather than ClusPro | CZSL model · UT-Zappos closed-world | AUC: 46.6 -> 48.3 | Helped |
| Apply TOMCAT rather than ClusPro | CZSL model · MIT-States closed-world | AUC: 23.8 -> 22.6 | Hurt |
| Apply TOMCAT rather than ClusPro | CZSL model · C-GQA closed-world | AUC: 14.9 -> 16.0 | Helped |
| Apply TOMCAT rather than AIF | CZSL model · Clothing16K closed-world | AUC: 98.2 -> 99.5 | Helped |
| Apply TOMCAT rather than ClusPro | CZSL model · UT-Zappos open-world | AUC: 39.5 -> 43.7 | Helped |
| Apply TOMCAT rather than ClusPro | CZSL model · MIT-States open-world | AUC: 9.3 -> 8.2 | Hurt |
| Apply TOMCAT rather than ClusPro | CZSL model · C-GQA open-world | AUC: 3.0 -> 4.2 | Helped |
| Apply TOMCAT rather than ATIF | CZSL model · Clothing16K open-world | AUC: 91.8 -> 95.8 | Helped |
| Add the priority queue and visual KAM to the base model (ablation) | TOMCAT · UT-Zappos closed-world | AUC: 43.57 -> 43.28 | Hurt |
| Add textual KAM only (ablation) | TOMCAT · UT-Zappos closed-world | AUC: 43.57 -> 45.68 | Helped |
| Add textual and visual KAMs and the queue, without adaptive update weights (ablation) | TOMCAT · UT-Zappos closed-world | AUC: 43.57 -> 46.64 | Helped |
| Add prediction entropy loss (ablation) | TOMCAT · UT-Zappos closed-world | AUC: 43.57 -> 44.59 | Helped |
Surprising findings
- Adding the priority queue and visual KAM alone degrades UT-Zappos performance versus the base model, attributed to uniform shoe styles.
- Using only the multimodal contrastive loss without entropy minimization degrades performance.
Limitations
- Knowledge accumulated only at composition level, ignoring primitive (attribute/object) information.
- Without proper priority queue initialization, the model is biased toward compositions of images already stored.
- Performance lags ClusPro on MIT-States, attributed to label noise.
Experimental setup
Datasets: UT-Zappos MIT-States C-GQA Clothing16K
Benchmarks: UT-Zappos MIT-States C-GQA Clothing16K
Metrics: AUC HM Seen accuracy Unseen accuracy
Baselines: CLIP CoOp Co-CGE CSP DFSP GIPCOL Troika CDS-CZSL PLID IMAX ClusPro online-TPS
Builds on: CLIP CoOp AdapterFormer Tip-Adapter TDA DPE
Closest papers by meaning
- A Dynamic Learning Method towards Realistic Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.91)
- Retrieval-Augmented Primitive Representations for Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.88)
- Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning (CVPR 2024 · similarity 0.88)
- Context-based and Diversity-driven Specificity in Compositional Zero-Shot Learning (CVPR 2024 · similarity 0.88)
- Revealing the Proximate Long-Tail Distribution in Compositional Zero-Shot Learning (AAAI 2024 · similarity 0.88)
- LOGICZSL: Exploring Logic-induced Representation for Compositional Zero-shot Learning (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.