Research map / Vision-language and multimodal / Human and embodied modeling
Whitened CLIP as a Likelihood Surrogate of Images and Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Whitened CLIP, a training-free linear whitening of CLIP embeddings so that squared norm gives a likelihood surrogate for images and captions, and applies it to artifact detection, domain shift, generator bias, and full-circle SLERP.
Problem
Computing explicit likelihoods for images (and captions) is hard for existing generative models, which provide scores or gradients rather than probabilities; CLIP's latent space has not been used as a likelihood estimator.
Approach
Apply a fixed, invertible PCA-based whitening to CLIP embeddings so they approximate an i.i.d. standard normal; the log-likelihood is then approximated by the negative squared Euclidean norm in the whitened space, computed once from a pre-computed whitening matrix.
Key results
- Whitened CLIP image embeddings pass normality tests for 98.3% (Anderson-Darling) and 99.3% (D'Agostino-Pearson) of features; text embeddings pass for 90.1% and 99.2%.
- On the nouns-removed task, W-CLIP reaches AUC 0.94 versus 0.43-0.69 for LLMs and VLMs (GPT2, OPT, NEO, BLIP, GIT).
- Opposite images from W-CLIP keep natural statistics (entropy 7.2 vs 7.3 for MS-COCO) while raw CLIP opposites collapse (entropy 4.8, 55.5% saturated pixels).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use W-CLIP likelihood scores instead of GIT scores to separate captions from general text | MS-COCO captions and OpenWebText · MS-COCO validation, captions vs. OpenWebText | AUC: 0.97 -> 0.999 | Helped |
| Use W-CLIP likelihood scores instead of GIT scores to detect captions with nouns removed | MS-COCO captions · MS-COCO validation, captions vs. captions with nouns removed | AUC: 0.69 -> 0.94 | Helped |
| Generate opposite images in W-CLIP rather than raw CLIP space | CLIP opposite-image generation · MS-COCO opposite images | Total Variation (TV): 156.7 -> 215.9 | Helped |
| Generate opposite images in W-CLIP rather than raw CLIP space | CLIP opposite-image generation · MS-COCO opposite images | Entropy: 4.8 -> 7.2 | Helped |
| Generate opposite images in W-CLIP rather than raw CLIP space | CLIP opposite-image generation · MS-COCO opposite images | Saturation pixels: 55.5 -> 6.4 | Helped |
| Use CLIP ViT-B/32 instead of CLIP ViT-L/14 (ablation) | W-CLIP · MS-COCO validation | Normality test scores | No clear effect |
| Compute the whitening matrix with more MS-COCO samples (ablation) | W-CLIP · MS-COCO validation embeddings | Normality test scores | Helped |
| Use at least 2k rather than 1k samples to compute the whitening matrix (ablation) | W-CLIP · MS-COCO validation embeddings, Anderson-Darling test | Normality | Helped |
| Normalize each iterative embedding to norm √d | UnCLIP iterative image generation · Iterative image generation | Image naturalness | Helped |
| Use W-CLIP rather than raw CLIP embeddings for full-circle SLERP | Full-circle SLERP · Image interpolation and extrapolation | Image naturalness | Helped |
| Add details or specificity to a caption | W-CLIP caption likelihood · Caption editing | Likelihood | Hurt |
| Remove details from a caption | W-CLIP caption likelihood · Caption editing | Likelihood | Helped |
Surprising findings
- Likelihood scores from W-CLIP correlate only moderately (0.33 to 0.48) with language-model log-likelihoods, indicating different information is captured.
- Iterative UnCLIP generation without norm normalization degrades into noise in the whitened space, and generated images from CLIP embeddings show a bias toward lower likelihood.
Limitations
- Whitening is sensitive to domain shift; the transform must be computed on a representative dataset.
- Evaluation is preliminary and mostly qualitative; likelihood is an approximation based on a Gaussian assumption and not a calibrated density.
- Correlated text features require replacing a feature with random noise, introducing randomness.
Experimental setup
Datasets: MS-COCO validation set Flickr8k OpenWebText SynArtifact ImageNet ImageNet-A ImageNet-C ImageNet-R
Benchmarks: MS-COCO validation Flickr8k SynArtifact ImageNet-A ImageNet-C ImageNet-R
Metrics: Anderson-Darling statistic D'Agostino-Pearson p-value Diagonal Score AUC Pearson correlation Total Variation Entropy saturation percentage
Baselines: GPT-2 OPT GPT-Neo BLIP GIT raw CLIP space
Builds on: CLIP UnCLIP SLERP
Closest papers by meaning
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.87)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.87)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.87)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.86)
- The Double-Ellipsoid Geometry of CLIP (ICML 2025 · similarity 0.86)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.