Research map / Vision-language and multimodal / CLIP and zero-shot
Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
Chuan Qin, Constantin Venhoff, Sonia Joseph, Fanyi Xiao, Stefan Scherer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train natively sparse CLIP (0.66% L0) vs dense | ViT-L/14 CLIP · zero-shot classification average | accuracy: 75.1 -> 75.6 | Improved |
| train natively sparse CLIP vs dense | ViT-L/14 CLIP · zero-shot fine-grained average | accuracy: 80.4 -> 81.0 | Improved |
| train natively sparse CLIP vs dense | ViT-L/14 CLIP · COCO BBox classification | Acc@1: 53.3 -> 55.5 | Improved |
| train natively sparse CLIP vs dense | ViT-L/14 CLIP · COCO retrieval | IR@1: 45.5 -> 43.7 | Hurt |
| lower logit scale cap from 50 to 40 (Sparse+) (ablation) | ViT-L/14 Sparse · zero-shot classification average | accuracy: 75.6 -> 75.1 | No clear effect |
| lower logit scale cap from 50 to 40 (Sparse+) (ablation) | ViT-L/14 Sparse · COCO retrieval | IR@1: 43.7 -> 41.8 | Hurt |
| native sparse training vs post-hoc SAE (Prisma) | OpenAI ViT-B/32 dense · zero-shot classification average | accuracy: 57.5 -> 56.6 | Hurt |
| add non-negativity (ReLU) with dimension expansion (ablation) | Sparse CLIP ViT-B/32 (15M pairs) · ImageNet-1k zero-shot | accuracy | Improved |
| use ReLU instead of L1 loss or TopK for sparsity (ablation) | Sparse CLIP ViT-B/32 (15M pairs) · ImageNet-1k zero-shot | accuracy | Improved |
| reduce logit scale cap to 20 (ablation) | Sparse CLIP ViT-B/32 (15M pairs) · ImageNet-1k zero-shot | accuracy | Hurt |
| use Sparse+ vision encoder in VLM | ViT-L/14 baseline VLM · AI2D | accuracy: 67.2 -> 70.6 | Improved |
| use Sparse+ vision encoder in VLM | ViT-L/14 baseline VLM · TextVQA | accuracy: 48.9 -> 48.5 | No clear effect |
Closest papers by meaning
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.93)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.92)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.91)
- Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder (NeurIPS 2026 · similarity 0.91)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.90)
- CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
