Research map / Vision-language and multimodal / CLIP and zero-shot
The Double-Ellipsoid Geometry of CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes the pre-normalization CLIP embedding (ViT-B/32) and shows image and text form separate, tilted, non-origin-centered thin-shell ellipsoids, then links this geometry to false negatives, conformity, the modality gap, and a training-free interpolation (vSLERP).
Problem
The geometry of CLIP's latent space is poorly understood; prior work mostly studies the unit-hypersphere embedding after L2 normalization, discarding norm and offset information.
Approach
Model the pre-normalized CLIP image and text embeddings as separable, tilted thin-shell ellipsoids shifted from the origin, and argue that this offset lets CLIP embed uncertain (frequent) concepts near the modality mean ("semantic blurring"), explaining the modality gap via conformity distribution matching.
Key results
- Estimated conformity (a·cos(m,v)+b) correlates with true conformity at Pearson 0.9998 for both image and text on MS-COCO.
- Image and text are linearly separable with 100% SVM accuracy using just two features (93 and 134) on MS-COCO.
- Conformity distributions of image and text match best near alpha≈0 (the trained setting), with KL≈0.14.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use features 93 and 134 to separate image and text embeddings | CLIP ViT-B/32 · MS-COCO val | linear SVM accuracy: 100% | Helped |
| Estimate conformity using cosine similarity to the modality mean | CLIP ViT-B/32 embeddings · MS-COCO | Pearson correlation with conformity: Pearson correlation of 0.9998 for both image and text | Helped |
| Shift the image ellipsoid toward the origin (ablation) | CLIP contrastive embeddings · MS-COCO val | combined contrastive loss: optimal α near zero | Hurt |
| Shift the image ellipsoid toward the origin (ablation) | CLIP contrastive embeddings · MS-COCO val, correctly classified samples | contrastive loss | Helped |
| Shift the image ellipsoid toward the origin (ablation) | CLIP contrastive embeddings · MS-COCO val, misclassified samples | contrastive loss | Hurt |
| Shift the text ellipsoid instead of the image ellipsoid (ablation) | CLIP contrastive embeddings · MS-COCO val | contrastive loss: optimal positions at α = 0 | Hurt |
| Shift both image and text ellipsoids toward the origin (ablation) | CLIP contrastive embeddings · MS-COCO val | contrastive loss: α = 1 | Hurt |
| Move the modality offsets away from their trained positions (ablation) | CLIP image-text conformity distributions · MS-COCO | KL divergence between conformity distributions: KL_{\alpha = 0} \approx 0.14 | Hurt |
Surprising findings
- Shifting the image ellipsoid to the origin lowers loss for correctly classified samples but raises it for misclassified ones, so the trained offset balances the two.
Limitations
- Analysis relies mainly on MS-COCO validation set (5k samples) as an approximation of the full training distribution.
- Thin-shell and approximate orthogonality assumptions underlie the conformity estimate proof.
- Main geometric claims are validated on ViT-B/32 with limited further model variants.
Experimental setup
Datasets: MS-COCO validation set ImageNet-A ImageNet-R
Benchmarks: MS-COCO validation set ImageNet-A ImageNet-R
Metrics: Pearson correlation KL divergence contrastive loss linear SVM accuracy cosine similarity
Baselines: unCLIP GLIDE ClipCap Caption Reward
Builds on: CLIP InfoNCE unCLIP SLERP
Closest papers by meaning
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.87)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.87)
- AmorLIP: Efficient Language-Image Pretraining via Amortization (NeurIPS 2025 · similarity 0.87)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.87)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.87)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.