Research map / Vision-language and multimodal / Multimodal LLMs
Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LEX-ICON, a multilingual mimetic-word and pseudo-word dataset, and probes MLLMs (Qwen2.5-Omni, Gemini-2.5-flash, GPT-4o) on sound symbolism across text, IPA, and audio inputs, with layer-wise phoneme attention analysis.
Problem
Whether multimodal LLMs associate phonetic forms with meanings non-arbitrarily, as humans do in sound symbolism (iconicity), across text, IPA and audio inputs.
Approach
Uses semantic-dimension binary A/B tests on natural mimetic words (4 languages) and systematically constructed CVCV pseudo-words, and measures phoneme-level attention fractions per layer to check whether models attend to sound-symbolic phonemes.
Key results
- MLLM macro-F1 averaged over input types exceeds the 0.50 baseline in 84.2% of semantic dimensions for natural words and 68.4% for constructed words.
- Qwen2.5-Omni-7B has the highest Pearson correlation with human evaluation (max r = 0.579).
- For constructed words, attention fraction on IPA text averages 0.523 vs 0.506 for audio, with iconic phonemes attended more in late layers.
Surprising findings
- Larger models such as gemini-2.5-flash deviate more from human response patterns than the smaller Qwen2.5-Omni-7B.
- Audio input does not consistently help; text input yields higher phoneme-level attention on iconic phonemes than audio for constructed words.
Limitations
- Ground truth for natural words is pseudo-labeled by LLM unanimity over dictionary definitions, and human evaluation covers only a sample of 10 non-linguist participants.
- Attention analysis is limited to Qwen2.5-Omni-7B due to access to open weights.
- Audio inputs are TTS-generated rather than natural speech; intonation is not studied.
Experimental setup
Datasets: LEX-ICON
Benchmarks: LEX-ICON
Metrics: macro-F1 accuracy Pearson correlation attention fraction score
Baselines: GPT-4o Gemini-2.5-flash Qwen2.5-Omni-3B
Builds on: semantic differential Montreal Forced Aligner Epitran
Closest papers by meaning
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.86)
- AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound (NeurIPS 2025 · similarity 0.85)
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.84)
- From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs (NeurIPS 2026 · similarity 0.84)
- VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation (ICLR 2026 · similarity 0.84)
- Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.