Research map / Vision-language and multimodal / Human and embodied modeling
Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CC:DAE, a complexity-constrained stochastic relaxation of Kolmogorov's structure function that measures conceptual similarity between images/texts via the AUC of a caption-length-dependent distance computed with LLaMA-13B and LLaVA.
Problem
Quantifying conceptual (semantic) similarity between images, and across text and images, in an objective, interpretable way that aligns with human judgments and avoids being dominated by incompressible randomness or discriminative features.
Approach
Define similarity as how well short natural-language descriptions of one sample fit the other as description complexity (coding length) grows; the distance is a KL-constrained (Lagrangian) optimal-description distribution evaluated by importance sampling, and the AUC over complexity gives a single score while the descriptions offer interpretability.
Key results
- On STS text-to-text benchmarks, CC:DAE averages 67.3 Spearman, beating zero-shot baselines including CLIP-ViT-L14 (67.0) and Meaning as Trajectories (65.3), though below SimCSE-BERT (76.3).
- On CxC-SIS image similarity, CC:DAE reaches 81.44 Spearman vs 72.08 for CLIP-ViT-B/16, a reported ~33.5% relative error reduction.
- On CxC-SITS cross-modal similarity, CC:DAE scores 67.71 vs 64.60 for CLIP-ViT-B/16 and comparable to Meaning as Trajectories (67.63).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CC:DAE instead of SimCSE-BERT | Sentence similarity evaluation · STS-B | Spearman correlation (×100): 68.4 -> 72.3 | Helped |
| Use CC:DAE instead of SimCSE-BERT | Sentence similarity evaluation · STS12 | Spearman correlation (×100): 82.4 -> 56.7 | Hurt |
| Use CC:DAE instead of SimCSE-BERT | Sentence similarity evaluation · STS13 | Spearman correlation (×100): 74.4 -> 67.9 | Hurt |
| Use CC:DAE instead of SimCSE-BERT | Sentence similarity evaluation · STS14 | Spearman correlation (×100): 80.9 -> 56.9 | Hurt |
| Use CC:DAE instead of SimCSE-BERT | Sentence similarity evaluation · STS15 | Spearman correlation (×100): 78.6 -> 68.7 | Hurt |
| Use CC:DAE instead of SimCSE-BERT | Sentence similarity evaluation · STS16 | Spearman correlation (×100): 76.9 -> 74.1 | Hurt |
| Use CC:DAE instead of SimCSE-BERT | Sentence similarity evaluation · SICK-R | Spearman correlation (×100): 72.2 -> 74.3 | Helped |
| Use CC:DAE instead of Meaning as Trajectories | Image similarity evaluation · CxC-SIS | Spearman correlation: 81.47 -> 81.44 | No clear effect |
| Use CC:DAE instead of Meaning as Trajectories | Image-text similarity evaluation · CxC-SITS | Spearman correlation: 67.63 -> 67.71 | No clear effect |
Surprising findings
- Conditional likelihood of text given the image is a poor similarity baseline on CxC-SITS (29.46), suggesting perplexity is a weak caption-alignment metric.
- Plain compression-based or structure-only measures do not capture semantics; the paper proves no non-trivial canonical notion of conceptual similarity exists without restricting descriptions.
Limitations
- Description space is limited to text, so similarity from visual arrangement and style is not efficiently captured.
- Results depend on non-canonical choices of language, encoder and prompt; no canonical notion of similarity exists.
- Image-to-image evaluation relies on a text bottleneck via captioning and a multimodal model.
Experimental setup
Datasets: STS12 STS13 STS14 STS15 STS16 STS-B SICK-R CxC-SIS CxC-SITS SugarCrepe
Benchmarks: STS-B STS12 STS13 STS14 STS15 STS16 SICK-R CxC-SIS CxC-SITS SugarCrepe
Metrics: Spearman correlation
Baselines: CLIP-ViT-L/14 SimCSE-BERT LLaMA conditional likelihood Meaning as Trajectories CLIP-ViT-B/16
Builds on: Kolmogorov Structure Function Meaning as Trajectories Normalized Compression Distance
Closest papers by meaning
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.89)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.87)
- Is Your Image a Good Storyteller? (AAAI 2025 · similarity 0.87)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.86)
- ConViS-Bench: Estimating Video Similarity Through Semantic Concepts (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.