Research map / Vision-language and multimodal / CLIP and zero-shot
Expediting Contrastive Language-Image Pretraining via Self-Distilled Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECLIPSE, a CLIP distillation framework where an online image encoder (EViT-sparsified) shares the text encoder with a momentum image teacher, matching soft image-text alignment matrices, and expedites training and inference.
Problem
Web-scraped image-text pairs are weakly correlated, causing data inefficiency in contrastive vision-language pretraining; existing distillation methods add costly image and text momentum encoders.
Approach
Replace the text momentum encoder with stop-gradient so that the online and momentum image encoders share one text encoder, distilling soft image-text alignment matrices within a unified projected space, and compensate the extra cost by token-sparsifying (EViT) the online image encoder.
Key results
- ECLIPSE with keep rate 0.5 achieves 18.37% ImageNet zero-shot top-1 on CC3M versus 16.06% for expedited CLIP, with 101% higher throughput (1132 vs 564 img/s at full ViT) than the full ViT.
- ECLIPSE without expedition (keep rate 1.0) reaches 20.32% vs CLIP's 17.10% on CC3M ImageNet zero-shot (+3.22).
- ECLIPSE at keep rate 0.7 gives 19.67% vs CLIP 17.10% with 54% faster throughput; on YFCC15M ECLIPSE (no extra supervision) averages 30.0 on 11 downstream datasets vs 27.5 for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ECLIPSE self-distillation instead of standard CLIP training | CLIP on CC3M · ImageNet-1k zero-shot | Top1 accuracy (%): 17.10 -> 19.67 | Helped |
| Apply EViT token sparsification without ECLIPSE | CLIP on CC3M · ImageNet-1k zero-shot | Top1 (%): 17.10 -> 16.55 | Hurt |
| Use ECLIPSE self-distillation instead of standard CLIP training | CLIP on YFCC15M · ImageNet-1k zero-shot | Top1 accuracy (%): 31.3 -> 32.7 | Helped |
| Use ECLIPSE self-distillation instead of standard CLIP training | CLIP on 88M · ImageNet-1k zero-shot | Top1 accuracy (%): 57.4 -> 60.2 | Helped |
| Add ECLIPSE distillation to hard-label training (ablation) | ECLIPSE on CC3M · ImageNet-1k zero-shot | Top1 (%): 16.1 -> 19.7 | Helped |
| Schedule λ progressively from 0.5 to 1 instead of keeping default λ=0.5 (ablation) | ECLIPSE on CC3M · ImageNet-1k zero-shot | Top1 (%): 19.7 -> 18.5 | Hurt |
| Distill alignment matrices instead of teacher image features (ablation) | ECLIPSE on CC3M · ImageNet-1k zero-shot | Top1 (%): 16.8 -> 19.7 | Helped |
| Use shared text embeddings with stop-gradient instead of a text momentum encoder (ablation) | ECLIPSE on CC3M · ImageNet-1k zero-shot | Top1 (%): 18.8 -> 19.7 | Helped |
| Keep fewer image tokens in the online encoder (ablation) | ECLIPSE on CC3M · ImageNet-1k zero-shot | Top1 accuracy (%): 20.32 (keep rate 1.0) -> 19.67 (keep rate 0.7); throughput 564 -> 871 | Mixed |
| Use augmented views for both online and momentum branches rather than the original image for the teacher (ablation) | ECLIPSE-ES on CC3M · ImageNet-1k zero-shot | Top1 accuracy (%): 23.91 -> 25.01 | Helped |
| Add efficient momentum-branch image self-supervision (ablation) | ECLIPSE-ES on CC3M · ImageNet-1k zero-shot | Top1 accuracy (%): 24.42 -> 25.01 | Helped |
Surprising findings
- Naively applying EViT token sparsification to CLIP, SLIP, and DeCLIP lowers zero-shot accuracy, whereas ECLIPSE with sparsification improves over its baseline.
- Conventional output-level feature distillation gives only minor gains for CLIP, and a text-momentum variant underperforms the stop-gradient design (e.g., 15.9 vs 19.7 top-1).
Limitations
- Relies on EViT token sparsification choice; a sweet spot in keep rate is needed, and aggressive sparsification degrades performance.
- The 88M curated dataset is not released, limiting reproducibility of the scaling results.
Experimental setup
Datasets: CC3M YFCC15M curated 88M image-text pairs ImageNet-1k Flickr30k MS-COCO Captions
Benchmarks: ImageNet-1k Pets CIFAR-10 CIFAR-100 SUN397 Food-101 Flowers Cars Caltech-101 Aircraft DTD Flickr30k
Metrics: top-1 zero-shot accuracy recall@1 recall@5 recall@10 throughput (images/s) training time (s/batch)
Baselines: CLIP CLIP w/ EViT SLIP SLIP w/ EViT DeCLIP DeCLIP w/ EViT ALBEF COTS FLIP
Builds on: CLIP SLIP DeCLIP EViT ALBEF SimSiam
Closest papers by meaning
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.88)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.87)
- FFF: Fixing Flawed Foundations in Contrastive Pre-Training Results in Very Strong Vision-Language Models (CVPR 2024 · similarity 0.87)
- Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder (NeurIPS 2026 · similarity 0.86)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.86)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.