Research map / Vision-language and multimodal / Human and embodied modeling
Investigating Inner Properties of Multimodal Representation and Semantic Compositionality With Brain-Based Componential Semantics
Shaonan Wang, Jiajun Zhang, Nan Lin, Chengqing Zong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Addition composition | Text adjective-noun composition · Text | Q1 rank: 5 | Improved |
| Use Matrix composition | Visual adjective-noun composition · Image | Q1 rank: 6 | Improved |
| Use Matrix composition | Multimodal adjective-noun composition · Multimodal | Q1 rank: 0 | Improved |
| Replace Addition with Multiplication composition | Text adjective-noun composition · Text | Q1 rank: 5 -> 1332 | Hurt |
| Use multimodal rather than single-modality phrase representations | Composition models · Multimodal | rank evaluation | Improved |
| Add visual input to linguistic word representations | Ridge multimodal representations · Brain-based sensory and motor properties | RSA correlation | Improved |
| Add visual input to linguistic word representations | Ridge multimodal representations · Brain-based abstract properties | RSA correlation | Hurt |
| Use MMskip multimodal representations instead of linguistic vectors | MMskip · Brain-based semantic properties | RSA correlation | No clear effect |
| Use visual input for abstract concepts | Multimodal representations · Abstract words | RSA correlation | Hurt |
| Use either linguistic or visual input for adjective-noun composition | Semantic composition · Adjective-noun phrases | Property difference pattern | No clear effect |
| Use parameterized composition models rather than Addition or Multiplication | Multimodal composition · 19 phrase categories | Mean property difference | Improved |
Closest papers by meaning
- The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities (ICLR 2025 · similarity 0.87)
- Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.86)
- Towards Neuron Attributions in Multi-Modal Large Language Models (NeurIPS 2024 · similarity 0.86)
- A Concept-Based Explainability Framework for Large Multimodal Models (NeurIPS 2024 · similarity 0.86)
- Debiased Multimodal Understanding for Human Language Sequences (AAAI 2025 · similarity 0.85)
- From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
