Research map / Vision-language and multimodal / CLIP and zero-shot
S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S3A, a teacher-student self-training framework with a Cluster-Vote-Prompt-Realign algorithm that adapts CLIP to realistic zero-shot classification over a 20K+ WordNet/ImageNet vocabulary without labels.
Problem
Zero-shot classification of unlabeled data when no ideal target vocabulary is given and only a huge (>20K class) open vocabulary is available, where CLIP accuracy degrades sharply due to confusing near-synonym candidates.
Approach
Iteratively clusters image embeddings, votes cluster-level class candidates from the large vocabulary, uses LLM-generated discriminative visual attributes to re-align clusters to words via Hungarian matching, and self-trains the CLIP image encoder with an EMA teacher using both structural (per-epoch CVPR) and individual (on-the-fly) pseudo-labels.
Key results
- S3A reaches 50.51% average top-1 accuracy over seven transductive benchmarks versus 35.29% for CLIP, an improvement of over 15 points.
- On the ImageNet-100 benchmark S3A achieves 52.08% top-1 and 82.76% clustering accuracy, versus 29.39% and 38.54% for CLIP.
- S3A outperforms adapted SCD and MUST by nearly 5% top-1 and 6% clustering accuracy on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use S³A rather than CLIP | CLIP ViT-B/16 · ImageNet-100 | top-1 accuracy: 29.39 -> 52.08 | Helped |
| Use S³A rather than the strongest baseline, MUST | CLIP ViT-B/16 · StanfordDogs | top-1 accuracy: 57.20 -> 58.94 | Helped |
| Add LLM-guided discriminative prompt augmentation (ablation) | S³A · ImageNet-100 | top-1 accuracy: 49.00 -> 52.08 | Helped |
| Add LLM-guided discriminative prompt augmentation (ablation) | S³A · Living17 | top-1 accuracy: 46.55 -> 48.34 | Helped |
| Add structural alignment to instance-wise self-training (ablation) | S³A · ImageNet-100 | clustering accuracy: 81.49 -> 82.08 | Helped |
| Add structural alignment to instance-wise self-training (ablation) | S³A · Living17 | clustering accuracy: 74.60 -> 73.04 | Hurt |
| Add self-training and structural alignment to the clustering-and-voting baseline (ablation) | S³A CVPR · ImageNet-100 | clustering accuracy: 77.39 -> 82.76 | Helped |
| Add self-training and structural alignment to the clustering-and-voting baseline (ablation) | S³A CVPR · Living17 | clustering accuracy: 68.81 -> 75.57 | Helped |
| Replace joint candidate prompting with single-category prompting (ablation) | S³A prompt augmentation · ImageNet-1K | top-1 accuracy: 37.69 -> 37.40 | Hurt |
| Use WordNet definitions for prompt augmentation (ablation) | S³A prompt augmentation · ImageNet-1K | top-1 accuracy: 37.95 -> 18.69 | Hurt |
| Use GPT-4 rather than ChatGPT for prompt augmentation (ablation) | S³A prompt augmentation · Entity13 | top-1 accuracy: 42.65 -> 44.98 | Helped |
| Use iterative rather than single-pass estimation of K (ablation) | S³A · Living17 | estimated cluster count: 52 -> 73 | Helped |
Surprising findings
- Naive instance-wise self-training (MUST-style) sometimes fails to improve over its initial representation when given the realistic large vocabulary.
- CLIP accuracy drops sharply when the vocabulary scales from the ideal set to 20K categories.
Limitations
- Requires the number of classes K to be known unless estimated; estimation is approximate and under-fragmentation hurts performance.
- Depends on LLM-generated attributes, which are costly and time-consuming to request.
- Relies on a fixed large vocabulary built from ImageNet-1K/21K and WordNet, so coverage of out-of-vocabulary categories is limited.
Experimental setup
Datasets: ImageNet-100 ImageNet-1K ImageNet-21K StanfordDogs Living17 Nonliving26 Entity13 Entity30 Oxford-IIIT Pet CIFAR100 Caltech101 WordNet
Benchmarks: ImageNet-100 ImageNet-1K StanfordDogs Living17 Nonliving26 Entity13 Entity30 Oxford-IIIT Pet CIFAR100 Caltech101
Metrics: top-1 accuracy clustering accuracy IoU soft accuracy (Sentence-BERT similarity)
Baselines: CLIP CLIP (Ideal) CLIP (Group) MUST SCD DINO+KMeans CHiLS BLIP-2
Builds on: CLIP SCD MUST GCD
Closest papers by meaning
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.89)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.89)
- Class Semantic Attribute Perception Guided Zero-Shot Learning (AAAI 2025 · similarity 0.88)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.88)
- ZeroMamba: Exploring Visual State Space Model for Zero-Shot Learning (AAAI 2025 · similarity 0.87)
- Simple Semantic-Aided Few-Shot Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.