Research map / Vision-language and multimodal / Cross-modal learning
Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-evaluates the bouba-kiki effect in CLIP (ResNet-50 and ViT) using label-probability preferences and Grad-CAM attention on shape-word matching, finding no consistent human-like association for pseudowords.
Problem
Whether vision-and-language models integrate cross-modal information in human-like ways, tested via the bouba-kiki effect (associating pseudowords like 'bouba' with round shapes and 'kiki' with jagged shapes), given mixed prior evidence.
Approach
Test CLIP ResNet and ViT with human-experiment-like designs using controlled generated curved/jagged image pairs and ten prompts, combining label-probability preference with a novel Grad-CAM attention analysis over concatenated image pairs, and compare directly with human data.
Key results
- Probability-based matching for pseudowords is mostly at chance level for both CLIP ResNet and ViT; only English adjectives are matched reliably (~80%).
- Grad-CAM-based correct matching reaches about 51.9% (sum of attention, ResNet) and 52.2% (ViT) per shape, with neither model reliably exceeding chance for paired bouba-kiki/maluma-takete labels.
- ResNet shows a general preference for round shapes, but overall behaviour falls short of human bouba-kiki performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace pseudowords with English adjectives | CLIP ResNet-50 and CLIP ViT · Shape-word matching | proportion correctly matched: roughly 80% | Helped |
| Use pseudowords instead of English adjectives | CLIP ResNet-50 and CLIP ViT · Shape-word matching | proportion congruent responses | Hurt |
| Evaluate shape-word preferences with Grad-CAM on pseudowords | CLIP ResNet-50 · Grad-CAM shape-word matching | cross-modal association | No clear effect |
| Evaluate shape-word preferences with Grad-CAM on pseudowords | CLIP ViT · Grad-CAM shape-word matching | cross-modal association | No clear effect |
| Use English adjectives instead of pseudowords in Grad-CAM evaluation | CLIP ViT · Grad-CAM shape-word matching | proportion correctly matched | Helped |
| Use entropy of attention instead of sum of attention (ablation) | CLIP ResNet-50 · Grad-CAM shape-word matching | correctness: .519 -> .488 | Hurt |
| Use centroid of attention instead of sum of attention (ablation) | CLIP ResNet-50 · Grad-CAM shape-word matching | correctness: .519 -> .515 | No clear effect |
| Use entropy of attention instead of sum of attention (ablation) | CLIP ViT · Grad-CAM shape-word matching | correctness: .522 -> .514 | No clear effect |
| Use centroid of attention instead of sum of attention (ablation) | CLIP ViT · Grad-CAM shape-word matching | correctness: .522 -> .515 | No clear effect |
Surprising findings
- Models attend mainly to shape centres or background artefacts rather than sharp or round contours.
- Learning about the bouba-kiki effect from data distributions does not translate into a mechanistic cross-modal preference.
Limitations
- Only CLIP ResNet and ViT tested; proprietary and more recent VLMs not evaluated.
- Text-only probing; audio (sound-symbolic) input not tested.
- Results depend on specific prompts, image sets, and pseudoword sets.
Experimental setup
Datasets: Images from Köhler-style and Nielsen and Rendall human studies Generated curved/jagged image pairs Pseudowords from Alper et al.
Benchmarks: bouba-kiki image-label matching maluma-takete matching
Metrics: proportion of congruent matches posterior mean with credible intervals (Bayesian regression) Grad-CAM attention sum correctness
Baselines: Alper et al. (2023) bouba-kiki VLM study Nielsen and Rendall (2011)-style human results English adjectives baseline
Builds on: Grad-CAM CLIP Alper and Lupyan-style sound-symbolism tests
Closest papers by meaning
- Can We Talk Models Into Seeing the World Differently? (ICLR 2025 · similarity 0.89)
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.88)
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.88)
- Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models (ICLR 2025 · similarity 0.88)
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.88)
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.