Research map / Vision-language and multimodal / CLIP and zero-shot
Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
David Méndez, Roberto Confalonieri, Natalia Díaz-Rodríguez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use cosine similarity to classifier weight rows instead of the linear layer with bias (ablation) | ImageNet-21K pretrained encoders · ImageNet-1K val | accuracy: 82.71 -> 82.10 (BEiT) | No clear effect |
| align with classifier head weights instead of averaged image representations of equal count (ablation) | MLP alignment BEiT-B/16 · Flickr30K zero-shot retrieval | i2t mAP: 0.1860 -> 0.2751 | Improved |
| align with classifier weights instead of one random image representation per class (ablation) | MLP alignment BEiT-B/16 · Flickr30K zero-shot retrieval | i2t mAP: 0.1311 -> 0.2751 | Improved |
| align with classifier weights instead of random image representations (ablation) | MLP alignment BEiT-B/16 · Zero-shot classification, 9 datasets | accuracy: CIFAR100 52.78 -> 61.12 | Improved |
| use ImageNet-21K head weights instead of ImageNet-1K weights (ablation) | MLP alignment BEiT-B/16 · Zero-shot classification, 9 datasets | accuracy | Improved |
| add one image-caption pair per class to ImageNet-21K weights (ablation) | MLP alignment BEiT-B/16 · Zero-shot classification, 9 datasets | accuracy | Mixed |
| use iNaturalist-2021 weights instead of ImageNet-21K weights (ablation) | MLP alignment BEiT-B/16 · Zero-shot classification benchmarks | accuracy | Hurt |
| augment image-text pairs with ImageNet-21K head weights (1K pairs) (ablation) | CSA · Flickr30K retrieval | i2t mAP: 0.0668 -> 0.4603 | Improved |
| augment image-text pairs with ImageNet-21K head weights (1K pairs) (ablation) | MLP alignment · Flickr30K retrieval | i2t mAP: 0.3702 -> 0.4945 | Improved |
| augment image-text pairs with ImageNet-21K head weights (30K pairs) (ablation) | Text2cpts · Flickr30K retrieval | i2t mAP: 0.5895 -> 0.5893 | No clear effect |
| augment image-text pairs with ImageNet-21K head weights (30K pairs) (ablation) | CSA · Flickr30K retrieval | t2i P@1: 0.2432 -> 0.2244 | Hurt |
| mitigate modality gap by centering/rescaling or lightweight projections (ablation) | weights plus image-text augmentation · downstream tasks | performance | No clear effect |
Closest papers by meaning
- S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment (AAAI 2024 · similarity 0.90)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.90)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.89)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
