AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training

Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, Lu Yuan

ECCV 2022 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 53 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add early specialization (conv stem / specialised first layer) (ablation)MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.99 -> 35.18Improved
add efficient parallel multi-scale CNN branch (ablation)MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.99 -> 34.18Improved
add both early specialization and parallel branch (ablation)MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.99 -> 36.66Improved
share transformer blocks across vision and text (ablation)CLIP (ViT-B/32, T768) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 31.85 -> 32.99Improved
widen text transformer from 512 to 768 (ablation)CLIP (ViT-B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.15 -> 31.85Hurt
also share the two LayerNorms across modalities (ablation)MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.99 -> 28.40Hurt
also share LN1 (LN before attention) (ablation)MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.99 -> 27.57Hurt
share LN2 in addition (LN1 stays specific) (ablation)MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.99 -> 32.16No clear effect
share all 12 layers instead of none (ablation)CLIP (ViT-B/32, T768) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 31.85 -> 32.99Improved
use NAS-searched layer sharing policy instead of sharing all layers (ablation)MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.99 -> 30.97Hurt
share only FFN instead of both Attn and FFN (ablation)MS-CLIP (B/32) · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 32.99 -> 30.40Hurt
use depth-wise conv in adapters instead of average pooling + FFN (ablation)MS-CLIP-S · ImageNet zero-shot (YFCC-22M, ViT-B/32)top-1 accuracy: 33.94 -> 36.66Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy