Research map / Vision-language and multimodal / CLIP and zero-shot
Building Vision-Language Models on Solid Foundations with Masked Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SF-CLIP, which trains a ViT-B/16 CLIP-style vision-language model with a contrastive loss plus masked feature distillation from frozen unimodal vision (SAM, DINOv2) and text (XGLM) teachers, applied to a random subset of each batch.
Problem
Contrastively trained vision-language models trained on noisy alt-text pairs lack spatial and compositional understanding, transfer poorly to dense vision tasks, and perform weakly in less common languages.
Approach
Combine CLIP contrastive loss with masked per-token feature distillation from frozen unimodal foundation teachers (SAM-H/16 and DINOv2-L/14 for vision, XGLM for text), apply distillation only to a small random subset of each minibatch for efficiency, and inherit frozen teacher word embeddings via a learned projection to gain multilingual ability without multilingual data.
Key results
- On YFCC-15M, SF-CLIP reaches 42.1 average zero-shot on ICinW vs 34.0 for CLIP and 40.1 for MaskCLIP.
- On CC-12M, LaSF-CLIP achieves 53.6 zero-shot ImageNet vs 48.4 for LaCLIP, and 46.9 ICinW average vs 41.9 for LaCLIP.
- SF-CLIP gets 79.0 EN and 48.7 ES I2T R@5 on XTD10 versus CLIP's 70.5 and 23.3, showing multilingual retrieval despite English-only training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add masked distillation and teacher word-embedding projection (ablation) | SF-CLIP on YFCC-15M · ImageNet-ZeroShot | accuracy: 31.5 -> 33.9 | Helped |
| add text teacher distillation (ablation) | SF-CLIP · XTD10 Spanish | image-to-text R@5: 18.4 -> 39.1 | Helped |
| add text teacher distillation (ablation) | SF-CLIP · XTD10 Russian | image-to-text R@5: 1.6 -> 34.2 | Helped |
| add text masking (ablation) | SF-CLIP · MS-COCO image-to-text retrieval | R@1: 32.5 -> 33.3 | Helped |
| add inherited teacher word-embedding projection (ablation) | SF-CLIP · ImageNet-ZeroShot | accuracy: 35.2 -> 33.9 | Hurt |
| add inherited teacher word-embedding projection (ablation) | SF-CLIP · XTD10 Russian | image-to-text R@5: 0.8 -> 34.2 | Helped |
| add image teacher distillation (ablation) | SF-CLIP · ImageNet-ZeroShot | accuracy: 31.3 -> 33.9 | Helped |
| add image teacher distillation (ablation) | SF-CLIP · Pascal-Context | zero-shot segmentation: 22.7 -> 25.4 | Helped |
| add image masking (ablation) | SF-CLIP · Pascal-Context | zero-shot segmentation: 25.4 -> 25.6 | No clear effect |
| reduce the distillation subset ratio to 6.25% (ablation) | SF-CLIP · ImageNet-ZeroShot | accuracy: 33.9 -> 33.9 | No clear effect |
| increase the distillation subset ratio to 25% (ablation) | SF-CLIP · ImageNet-ZeroShot | accuracy: 33.9 -> 34.1 | No clear effect |
| increase text teacher size from 564M to 1.7B (ablation) | SF-CLIP with DINO vision teacher · ImageNet-ZeroShot | accuracy: 33.9 -> 34.6 | Helped |
Surprising findings
- Multilingual retrieval emerges from inheriting frozen XGLM word embeddings and distilling a multilingual LLM, even though no non-English paired data is used.
- Image masking gave mixed results for ViT-B, so it was disabled by default.
Limitations
- Requires the same tokenizer as the text teacher, which is restrictive.
- Cannot be trivially applied to fine-tune an existing pretrained CLIP model.
- Uses linear projections instead of transformer decoders for distillation, possibly limiting learning capacity.
Experimental setup
Datasets: YFCC15M CC12M ImageNet-1k Flickr30K MS-COCO ICinW SugarCREPE SVO XTD10 Pascal-Context ADE-20K Pascal-VOC
Benchmarks: ICinW ImageNet-1k Flickr30K MS-COCO Pascal-Context ADE-20K Pascal-VOC COCO-Stuff SugarCREPE SVO XTD10
Metrics: zero-shot accuracy linear probe accuracy recall@K mIoU mAP mAR
Baselines: CLIP SLIP MaskCLIP LaCLIP SLIP100ep LaSLIP
Builds on: CLIP SLIP LaCLIP MaskCLIP OpenCLIP
Closest papers by meaning
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.91)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.91)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · similarity 0.90)
- Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling (NeurIPS 2026 · similarity 0.90)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.