Research map / Label-efficient and robust learning / Recognition and long-tail
Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpLiCE, a training-free, task-agnostic method that decomposes dense CLIP embeddings into sparse nonnegative combinations of a LAION-derived 1- and 2-word concept vocabulary via a lasso-style sparse recovery with modality alignment.
Problem
CLIP embeddings are dense and uninterpretable, making it hard to know what semantic content they encode and to use them transparently in downstream applications.
Approach
Approximate CLIP image embeddings as sparse, nonnegative linear combinations of a large overcomplete, task-agnostic dictionary of CLIP text embeddings of common LAION words/bigrams, after mean-centering image and text embeddings to align modalities; solved with an l1-relaxed nonnegative lasso, requiring no training.
Key results
- SpLiCE with ~20-35 nonzero concepts reaches 0.86 Caltech101, 0.66 SUN397, 0.96 STL10, 0.83 VOC2007 zero-shot accuracy, close to CLIP-reported values.
- At sparsity 512, SpLiCE matches or exceeds CLIP zero-shot accuracy (CIFAR100 0.768 vs 0.750; MIT States 0.474 vs 0.469; ImageNet 0.552 vs 0.552).
- In a CelebA glasses intervention, zeroing the glasses concept drops glasses zero-shot accuracy from 0.91 to 0.69 while gender accuracy stays near 0.97-0.96; on Waterbirds, worst-group accuracy rises from 0.48 to 0.60.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SpLiCE with sparsity 20-35 instead of dense CLIP | OpenCLIP ViT-B/32 zero-shot classification · Caltech101 | zero-shot accuracy: 0.90 -> 0.86 | Hurt |
| Use SpLiCE with sparsity 20-35 instead of dense CLIP | OpenCLIP ViT-B/32 zero-shot classification · SUN397 | zero-shot accuracy: 0.67 -> 0.66 | Hurt |
| Use SpLiCE with sparsity 20-35 instead of dense CLIP | OpenCLIP ViT-B/32 zero-shot classification · STL10 | zero-shot accuracy: 0.96 -> 0.96 | No clear effect |
| Use SpLiCE with sparsity 20-35 instead of dense CLIP | OpenCLIP ViT-B/32 zero-shot classification · VOC 2007 | zero-shot accuracy: 0.92 -> 0.83 | Hurt |
| Increase SpLiCE decomposition sparsity to 512 (ablation) | SpLiCE · CIFAR100 | zero-shot accuracy: 0.750 -> 0.768 | Helped |
| Zero out the eyewear/glasses concept in SpLiCE representations (ablation) | ZS SpLiCE · CelebA | zero-shot accuracy (Glasses): 0.88 -> 0.69 | Hurt |
| Zero out the eyewear/glasses concept in SpLiCE representations (ablation) | ZS SpLiCE · CelebA | zero-shot accuracy (Gender): 0.97 -> 0.96 | Hurt |
| Train the probe on SpLiCE rather than dense CLIP embeddings | CIFAR100 linear probing · CIFAR100 | probe accuracy: 0.97 -> 0.95 | Hurt |
| Train the probe on SpLiCE rather than dense CLIP embeddings | MIT States linear probing · MIT States | probe accuracy: 0.883 -> 0.883 | No clear effect |
| Replace the LAION task-agnostic dictionary with an LLM-generated concept dictionary (ablation) | SpLiCE · CIFAR100 | zero-shot accuracy | Hurt |
| Replace the LAION task-agnostic dictionary with an LLM-generated concept dictionary (ablation) | SpLiCE · CIFAR100 | cosine reconstruction | Hurt |
| Allow negative concept weights (ablation) | SpLiCE · CIFAR100 | cosine reconstruction | Helped |
Surprising findings
- SpLiCE's cosine reconstruction of CLIP embeddings is low (around the level of random dictionaries) even though zero-shot accuracy is nearly preserved, indicating it captures semantics but not non-semantic components.
- Antonym concept pairs are not opposite in CLIP space (average cosine similarity 0.72 without centering), so negative concepts are not reliable.
Limitations
- Concept vocabulary limited to one- and two-word phrases, so longer concepts are not captured.
- Vocabulary derived from LAION token frequency, which carries its own biases and may miss salient CLIP concepts.
- Uses an l1 penalty as a relaxation of the l0 constraint, which may not give the sparsest solutions.
Experimental setup
Datasets: LAION-400m captions (concept vocabulary) MSCOCO (modality alignment mean, evaluation) CIFAR100 MIT States CelebA ImageNet validation Waterbirds Stanford Cars
Benchmarks: CIFAR100 MIT States ImageNet MSCOCO CelebA Caltech101 SUN397 STL10 VOC2007 Waterbirds
Metrics: zero-shot accuracy cosine similarity reconstruction linear probe accuracy retrieval recall@k semantic relevance (Hausdorff distance) user study Likert ratings worst-group accuracy
Baselines: LF-CBM IP-OMP random dictionary learned dictionary (FISTA) LLM-generated concept dictionary
Builds on: CLIP linear representation hypothesis modality gap concept bottleneck models LF-CBM
Closest papers by meaning
- Interpreting CLIP with Hierarchical Sparse Autoencoders (ICML 2025 · similarity 0.89)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.87)
- Interpreting CLIP's Image Representation via Text-Based Decomposition (ICLR 2024 · similarity 0.86)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.86)
- Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP (NeurIPS 2024 · similarity 0.86)
- Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.