Research map / Vision-language and multimodal / Human and embodied modeling
Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that visual embeddings of pre-trained VLMs (CLIP) have approximate compositional structure and proposes training-free Geodesically Decomposable Embeddings (GDE), evaluated on compositional classification and group robustness.
Problem
Whether compositional structure emerges in the visual embedding space of vision-language models, given noise and sparsity of image data, unlike text embeddings studied previously.
Approach
Extends linear compositional decomposition of text embeddings (ideal words) to Riemannian manifolds: image embeddings are denoised via weighted intrinsic means and tangent-space vector means, with per-primitive directions combined via exponential map, handling noise via CLIP image-to-text probabilities and sparsity via available tuples.
Key results
- On UT-Zappos closed-world, GDE (image) reaches AUC 13.9 vs 4.4 for CLIP (ρ 317.9%) and far above LDE (image) at 0.9.
- On Waterbirds, GDE achieves 86.4 worst-group accuracy with a gap of 5.0, outperforming FairerCLIP (86.0 WG, gap 6.1) and CLIP (44.4 WG).
- On CelebA, GDE gets 87.5 WG and a gap of 0.4, versus 85.2 WG for FairerCLIP; GDE keeps WG within 1% when using 25% of training samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use geodesic rather than linear decomposition for image embeddings | GDE compositional classification · UT-Zappos | AUC: 0.9 -> 13.9 | Helped |
| use geodesic rather than linear decomposition for image embeddings | GDE compositional classification · MIT-States | AUC: 2.0 -> 8.6 | Helped |
| replace the CLIP zero-shot full-state embeddings with GDE image embeddings | compositional classification · UT-Zappos closed-world | AUC: 4.4 -> 13.9 | Helped |
| replace the CLIP zero-shot full-state embeddings with GDE image embeddings | compositional classification · MIT-States closed-world | AUC: 11.1 -> 8.6 | Hurt |
| replace the CLIP zero-shot full-state embeddings with GDE image embeddings | compositional classification · MIT-States open-world | AUC: 1.7 -> 1.8 | Helped |
| use GDE image embeddings instead of CLIP zero-shot embeddings | group robustness · Waterbirds | WG accuracy: 44.4 -> 86.4 | Helped |
| use GDE image embeddings instead of FairerCLIP | group robustness · Waterbirds | WG accuracy: 86.0 -> 86.4 | Helped |
| use GDE image embeddings instead of FairerCLIP | group robustness · CelebA | WG accuracy: 85.2 -> 87.5 | Helped |
| use CLIP RN50 instead of CLIP ViT-L/14 as the backbone (ablation) | GDE compositional classification · UT-Zappos closed-world | AUC: 13.9 -> 8.6 | Hurt |
| use SigLIP ViT-SO400M/14 instead of CLIP ViT-L/14 as the backbone (ablation) | GDE compositional classification · UT-Zappos closed-world | AUC: 13.9 -> 24.7 | Helped |
| use CLIP RN50 instead of CLIP ViT-L/14 as the backbone (ablation) | GDE compositional classification · MIT-States closed-world | AUC: 8.6 -> 3.4 | Hurt |
| use SigLIP ViT-SO400M/14 instead of CLIP ViT-L/14 as the backbone (ablation) | GDE compositional classification · MIT-States closed-world | AUC: 8.6 -> 11.9 | Helped |
Surprising findings
- Linear decomposition (LDE) of image embeddings performs much worse than CLIP zero-shot, indicating image embeddings are not closely linearly decomposable.
- Decomposed image embeddings used with unCLIP diffusion produce plausible images of unseen attribute-object pairs and blended animal species.
Limitations
- Visual decomposition relies on noisy and sparse image sets, requiring denoising via CLIP probabilities and a tuned temperature.
- Evaluated on a limited set of attribute-object datasets with two-factor compositions.
Experimental setup
Datasets: UT-Zappos MIT-states Waterbirds CelebA
Benchmarks: UT-Zappos MIT-states Waterbirds CelebA
Metrics: ATTR OBJ SEEN UNSEEN HM AUC worst-group accuracy average accuracy GAP
Baselines: CLIP zero-shot LDE (linearly decomposable embeddings) ERM linear probe ERM adapter DFR Contrastive Adapters FairerCLIP
Builds on: ideal words (Trager et al.) Principal Geodesic Analysis CLIP FairerCLIP
Closest papers by meaning
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.91)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.90)
- Beyond the Linear Separability Ceiling: Aligning Representations in VLMs (NeurIPS 2026 · similarity 0.90)
- CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.90)
- Compositional Entailment Learning for Hyperbolic Vision-Language Models (ICLR 2025 · similarity 0.90)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.