Research map / Vision-language and multimodal / Cross-modal learning
MSAmba: Exploring Multimodal Sentiment Analysis with State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSAmba, a hybrid Mamba architecture (intra-modal sequential Mamba blocks plus cross-modal hybrid Mamba-attention blocks) for multimodal sentiment regression on CMU-MOSI, CMU-MOSEI and CH-SIMS.
Problem
Multimodal sentiment analysis requires modeling sequential intra-modal information and cross-modal interactions, but Transformer- and CNN-based methods have many parameters and slow inference; Mamba has not been adapted to MSA.
Approach
Replaces Transformer-based sequence modeling and cross-attention with Mamba: an ISM block with global-local context extraction and bi-directional selective scanning for each modality, and a CHM block that injects a centralized (language) modality into the other modalities via Mamba plus self-attention hybrid layers, trained with L1 loss and auxiliary losses.
Key results
- On CMU-MOSEI, MSAmba reaches 85.99 F1 (negative/positive) and 85.78 Acc-2, surpassing ALMT's Acc-2 by 1.00%.
- On CMU-MOSI, MSAmba achieves Acc-7 of 49.67 with 1.41M parameters, versus ALMT's 49.42 with 2.49M parameters.
- On CH-SIMS, MSAmba obtains 82.30 Acc-2 and 0.403 MAE, outperforming ALMT (81.19 Acc-2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MSAmba instead of ALMT | MSAmba · CMU-MOSI | Acc-7: 49.42 -> 49.67 | Helped |
| Use MSAmba instead of ConFEDE | MSAmba · CMU-MOSEI | MAE: 0.522 -> 0.507 | Helped |
| Use MSAmba instead of ConFEDE | MSAmba · CH-SIMS | Acc-2: 81.05 -> 82.30 | Helped |
| Add self-attention to CHM (ablation) | MSAmba · CMU-MOSI | Acc-7: 47.83 -> 49.67 | Helped |
| Add GLCE to ISM (ablation) | MSAmba · CMU-MOSI | Acc-7: 48.92 -> 49.67 | Helped |
| Use bidirectional BSSM instead of unidirectional SSM (ablation) | MSAmba · CMU-MOSI | Acc-7: 46.57 -> 49.67 | Helped |
| Use ISM instead of Vision Mamba (ablation) | MSAmba · CMU-MOSI | Acc-7: 47.01 -> 49.67 | Helped |
| Use ISM instead of Video Mamba (ablation) | MSAmba · CMU-MOSI | Acc-7: 47.18 -> 49.67 | Helped |
| Use ISM instead of a Transformer for sequential modeling (ablation) | MSAmba · CMU-MOSI | Acc-7: 47.67 -> 49.67 | Helped |
| Use hybrid CHM instead of pure SSM in cross-modal interaction (ablation) | MSAmba · CMU-MOSI | Acc-7: 46.29 -> 49.67 | Helped |
| Use hybrid CHM instead of cross-attention Transformer (ablation) | MSAmba · CMU-MOSI | Acc-7: 47.81 -> 49.67 | Helped |
Surprising findings
- Simply applying vanilla Mamba blocks naively achieves results comparable to previous SOTAs, per the authors.
- Adding self-attention inside CHM increases overfitting on small datasets, per the authors' hyperparameter study.
Limitations
- Self-attention in CHM increases parameters and overfitting on limited datasets.
- Results are reported on only three benchmark datasets with a single-GPU setup.
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI CH-SIMS
Benchmarks: CMU-MOSI CMU-MOSEI CH-SIMS
Metrics: Acc-2 F1 Acc-7 Acc-5 Acc-3 MAE Corr
Baselines: MFM LMF TFN MuLT MISA MAG-BERT Self-MM ConFEDE ALMT CLGSI
Builds on: Mamba Video Mamba ALMT S4
Closest papers by meaning
- MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition (AAAI 2025 · similarity 0.88)
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.87)
- Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.87)
- A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.86)
- SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.86)
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.