Research map / Vision-language and multimodal / CLIP and zero-shot
Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training
Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, Lu Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add early specialization (conv stem / specialised first layer) (ablation) | MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.99 -> 35.18 | Improved |
| add efficient parallel multi-scale CNN branch (ablation) | MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.99 -> 34.18 | Improved |
| add both early specialization and parallel branch (ablation) | MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.99 -> 36.66 | Improved |
| share transformer blocks across vision and text (ablation) | CLIP (ViT-B/32, T768) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 31.85 -> 32.99 | Improved |
| widen text transformer from 512 to 768 (ablation) | CLIP (ViT-B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.15 -> 31.85 | Hurt |
| also share the two LayerNorms across modalities (ablation) | MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.99 -> 28.40 | Hurt |
| also share LN1 (LN before attention) (ablation) | MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.99 -> 27.57 | Hurt |
| share LN2 in addition (LN1 stays specific) (ablation) | MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.99 -> 32.16 | No clear effect |
| share all 12 layers instead of none (ablation) | CLIP (ViT-B/32, T768) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 31.85 -> 32.99 | Improved |
| use NAS-searched layer sharing policy instead of sharing all layers (ablation) | MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.99 -> 30.97 | Hurt |
| share only FFN instead of both Attn and FFN (ablation) | MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 32.99 -> 30.40 | Hurt |
| use depth-wise conv in adapters instead of average pooling + FFN (ablation) | MS-CLIP-S · ImageNet zero-shot (YFCC-22M, ViT-B/32) | top-1 accuracy: 33.94 -> 36.66 | Improved |
Closest papers by meaning
- Contrastive Vision-Language Pre-training with Limited Resources (ECCV 2022 · similarity 0.92)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.91)
- COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training (CVPR 2025 · similarity 0.89)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.89)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.89)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
