Research map / Vision-language and multimodal / Cross-modal learning
Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Coupled Mamba, a state-space model that couples per-modality Mamba state chains via summed inter-modal state transitions with a global convolution kernel for parallel training, evaluated on multimodal sentiment analysis (CMU-MOSEI, CH-SIMS, CH-SIMSV2).
Problem
Multimodal fusion with state space models is under-explored; existing SSMs (Mamba) process single modalities, and naive fusion neglects intra-modal dynamics or inter-modal interactions, while preserving hardware-parallel computation.
Approach
Couples the hidden-state chains of multiple modalities by summing all modalities' previous states before a shared state transition (inspired by Coupled HMMs), and derives a multi-modal global convolution kernel so the coupled recurrence remains parallelizable like Mamba.
Key results
- On CMU-MOSEI regression, Coupled Mamba reports MAE 0.547, Corr 0.756 and Acc-2 85.6 / F1 85.5 on unaligned data, versus best prior unaligned TETFN at 85.1/85.2.
- On CH-SIMS, Coupled Mamba reaches Acc-2 81.8 and F1 81.3 versus 81.2 and 80.4 for the best prior baselines.
- The paper reports up to 83.7% GPU memory savings at sequence length 500 and about 49% faster inference than Transformers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Coupled Mamba fusion instead of cross-attention (ablation) | CMU-MOSEI unaligned · CMU-MOSEI | F1-Score: 84.5 -> 85.5 | Helped |
| use coupled fusion instead of average fusion (ablation) | CMU-MOSEI · CMU-MOSEI | F1-Score: 84.1 -> 85.5 | Helped |
| use coupled fusion instead of concatenation fusion (ablation) | CMU-MOSEI · CMU-MOSEI | F1-Score: 84.5 -> 85.5 | Helped |
| add state coupling to Mamba fusion (ablation) | CMU-MOSEI · CMU-MOSEI | F1-Score: 85.3 -> 85.5 | Helped |
| set Δ to dstate/16 rather than dstate/8 (ablation) | Coupled Mamba · CMU-MOSEI | F1-Score: 85.5 -> 85.0 | Hurt |
| set Δ to dstate/4 rather than dstate/8 (ablation) | Coupled Mamba · CMU-MOSEI | F1-Score: 85.5 -> 84.9 | Hurt |
| increase dstate from 64 to 128 (ablation) | Coupled Mamba · CMU-MOSEI | F1-Score: 85.5 -> 84.1 | Hurt |
| reduce dstate from 64 to 32 (ablation) | Coupled Mamba · CMU-MOSEI | F1-Score: 85.5 -> 84.9 | Hurt |
| use unaligned rather than aligned input | Coupled Mamba · CMU-MOSEI | F1-Score: 85.6 vs 85.5 | No clear effect |
| use Coupled Mamba rather than the strongest reported baseline on CH-SIMS | CH-SIMS unaligned · CH-SIMS | F1-Score: 80.4 -> 81.3 | Helped |
| use Coupled Mamba rather than the strongest reported baseline on CH-SIMSV2 | CH-SIMSV2 unaligned · CH-SIMSV2 | F1-Score | Helped |
Surprising findings
- Cross-attention fusion underperforms Coupled Mamba by about 1 F1 point on CMU-MOSEI, which the authors attribute to misaligned unaligned data.
Limitations
- Requires all modalities to share the same state size, addressed only by projection layers suggested but not evaluated.
- Evaluated only on sentiment analysis datasets with BERT text features.
- Reported gains over baselines are small (e.g., 0.4 to 0.9 F1 points on CMU-MOSEI and CH-SIMS) and single-run variance is not reported beyond averages.
Experimental setup
Datasets: CMU-MOSEI CH-SIMS CH-SIMSV2
Benchmarks: CMU-MOSEI CH-SIMS CH-SIMSV2
Metrics: MAE Corr Acc-2 Acc-3 Acc-5 F1-Score F1-Score-3
Baselines: TFN LMF MFN MFM MulT MAG-BERT ICCN MISA TETFN DMD IMDer Self-MM
Builds on: Mamba S4 Coupled Hidden Markov Model
Closest papers by meaning
- AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.87)
- EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality (CVPR 2025 · similarity 0.87)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.85)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.85)
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data (NeurIPS 2025 · similarity 0.85)
- Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.