Research map / Vision-language and multimodal / CLIP and zero-shot
Super-Class Guided Transformer for Zero-Shot Attribute Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SugaFormer, a super-class guided transformer that uses super-class queries, multi-context decoding, and VLM-based consistency regularization and retrieval score enhancement for zero-shot attribute classification.
Problem
Zero-shot attribute classification suffers from poor generalization to unseen attributes and poor scalability of class-wise queries when the number of attribute classes is large.
Approach
Replace per-class decoder queries with shared super-class queries initialized from super-class text embeddings and object-conditioned visual features, decode with three visual contexts (crop, mask, image), and transfer VLM knowledge via a super-class guided prompt consistency loss in training and a Q-Former retrieval score added to Top-K unseen predictions at inference.
Key results
- On VAW zero-shot, SugaFormer reaches AP_novel 60.59 and AP_all 73.32, versus 53.38 and 70.61 for the ML-Decoder baseline.
- On OVAD cross-dataset transfer, SugaFormer achieves AP_all 29.7, above OvarNet's 28.6.
- On LSA zero-shot, SugaFormer with 9 queries achieves AP_novel 5.80, while the class-wise ML-Decoder runs out of memory with 9538 queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SugaFormer instead of ML-Decoder | BLIP2-based attribute classification · VAW zero-shot | AP_novel: 53.38 -> 60.59 | Helped |
| Apply SugaFormer instead of OvarNet | attribute classification · LSA zero-shot | AP_novel: 5.48 -> 5.80 | Helped |
| Apply SugaFormer instead of OvarNet | attribute classification trained on VAW · OVAD cross-dataset transfer | AP_all: 28.6 -> 29.7 | Helped |
| Add super-class query initialization (SQI) (ablation) | ML-Decoder · VAW zero-shot | AP_novel: 53.38 -> 55.90 | Helped |
| Add multi-context decoding (MD) (ablation) | SQI · VAW zero-shot | AP_all: 70.67 -> 72.07 | Helped |
| Add super-class guided consistency regularization (SCR) (ablation) | SQI + MD · VAW zero-shot | AP_novel: 56.25 -> 58.38 | Helped |
| Add zero-shot retrieval-based score enhancement (ZRSE) (ablation) | SQI + MD + SCR · VAW zero-shot | AP_novel: 58.38 -> 60.59 | Helped |
| Use a super-class guided prompt instead of a general prompt (ablation) | SCR · VAW zero-shot | AP_novel: 55.64 -> 60.59 | Helped |
| Use both attribute-class and object-category embeddings for SQI pooling (ablation) | SQI · VAW zero-shot | AP_novel: 59.40 -> 60.59 | Helped |
| Apply SCR and ZRSE (ablation) | UPT · HICO-DET RF-UC | Unseen mAP: 23.55 -> 26.45 (+2.90) | Helped |
| Apply SCR and ZRSE (ablation) | UPT · HICO-DET UV | Unseen mAP: 19.25 -> 22.19 (+2.94) | Helped |
| Apply SCR and ZRSE (ablation) | PViC · HICO-DET RF-UC | Unseen mAP: 25.69 -> 28.16 (+2.47) | Helped |
Surprising findings
- Training on VAW alone transfers poorly to OVAD compared with image-caption datasets, as noted in the OVAD work.
Limitations
- Super-class mapping may be suboptimal; better mapping methods could improve performance.
- Multiple attribute classes in the same super-class share one query output, weakening the learning signal for distinguishing them.
Experimental setup
Datasets: VAW LSA OVAD CC-3M COCO-Captions
Benchmarks: VAW LSA OVAD HICO-DET
Metrics: average precision mean average precision
Baselines: CLIP BLIP2 CLIP-Attr OvarNet ML-Decoder TAP ALBEF BLIP X-VLM OVAD
Builds on: ML-Decoder BLIP-2 OvarNet asymmetric loss
Closest papers by meaning
- Class Semantic Attribute Perception Guided Zero-Shot Learning (AAAI 2025 · similarity 0.88)
- Label Propagation for Zero-shot Classification with Vision-Language Models (CVPR 2024 · similarity 0.88)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.88)
- Classification Done Right for Vision-Language Pre-Training (NeurIPS 2024 · similarity 0.87)
- S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment (AAAI 2024 · similarity 0.87)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.