Research map / Vision-language and multimodal / Multimodal LLMs

Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a logic-based synthetic framework with six modality interaction types to diagnose when additional modalities help or hurt multimodal reasoning in MLLMs, and probes internals to identify task-composition and fusion bottlenecks.

Problem

It is unclear when and why adding modalities (vision, audio) helps or harms reasoning in MLLMs, since prior evaluations are anecdotal and do not control where decisive facts lie or how they must be combined.

Approach

A controlled logic-based evaluation with six canonical interaction types (equivalence, alternative, entailment, independence, contradictory, complementary) built from synthetic facts rendered as text, TTS audio, and Graphviz images; combined with attention probing and layer-wise temperature interventions to locate the bottlenecks.

Key results

• Alternative interaction improves multimodal accuracy over text-only on average (98.7% vs 97.0% text), while Entailment drops accuracy by 7.8-12.8 points across modalities.
• Independence multimodal accuracy averages 70.3% vs 94.5% text-only unimodal, and Complementary multimodal averages 52.0%, below every unimodal condition.
• Linear probes recover modality identity perfectly from attention features, and raising softmax temperature in the first four decoder layers improves reasoning, while middle/late-layer changes have little effect.

Surprising findings

• Adding equivalent facts in a text-redundant setting reduces accuracy for some models (e.g., Baichuan -11.1 with text), i.e., redundancy can hurt.
• Attention patterns do not reliably encode fact usefulness, despite models recognizing facts nearly perfectly.

Limitations

• Synthetic, single-step logic tasks with schematic renderings may not transfer to natural multimodal reasoning; IsoBench validation is only partial (Equivalence setting only).
• Only four open-source MLLMs evaluated, all around 5-8B parameters.
• Interventions are shallow (temperature scaling and two-step prompting) and not shown to fully resolve the bottlenecks.

Experimental setup

Datasets: synthetic multimodal logical reasoning instances (text, TTS audio, Graphviz images) IsoBench

Benchmarks: OmniReason (synthetic six-interaction set) IsoBench

Metrics: accuracy accuracy delta vs unimodal answer selection ratio linear probe accuracy

Baselines: Baichuan-Omni-1.5 Qwen2.5-Omni MiniCPM-o-2.6 Phi-4 Multimodal

Builds on: HELM synthetic reasoning

Closest papers by meaning

• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.92)
• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.91)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.91)
• Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.91)
• Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark (ICML 2025 · similarity 0.91)
• Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models (CVPR 2026 · similarity 0.91)