Research map / Vision-language and multimodal / Cross-modal learning
AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AlignMamba, which adds OT-based token-level local alignment and MMD-based global alignment before a Mamba backbone to fuse audio, video, and language sequences efficiently.
Problem
Mamba's sequential scanning struggles to model cross-modal correspondences, especially between unscanned tokens, while Transformer-based multimodal fusion has quadratic cost on long sequences.
Approach
Align unimodal token sequences with a relaxed optimal transport plan (cosine cost) for local token-level correspondence and an MMD loss with Gaussian kernel for global distribution alignment, then fuse via temporally interleaved sequence in a Mamba backbone.
Key results
- On CMU-MOSI, AlignMamba reaches 86.9% accuracy / 86.9 F1 on complete fusion, a 0.9-point gain over the best prior method (DMD and MTMD at 86.0 or 85.8 in the table range).
- With 6.4k tokens, AlignMamba uses 8.53 GB GPU memory (20.3% less than single-stream Transformer) and 6.05 s inference (83.3% less than single-stream Transformer).
- Removing both alignment modules drops CMU-MOSI accuracy to 82.5%, versus 86.9% for the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OT-based local alignment (ablation) | AlignMamba · CMU-MOSI | Accuracy: 84.6 -> 86.9 | Helped |
| Add OT-based local alignment (ablation) | AlignMamba · CMU-MOSEI | Accuracy: 84.1 -> 86.6 | Helped |
| Add MMD-based global alignment (ablation) | AlignMamba · CMU-MOSI | Accuracy: 85.8 -> 86.9 | Helped |
| Add MMD-based global alignment (ablation) | AlignMamba · CMU-MOSEI | Accuracy: 85.7 -> 86.6 | Helped |
| Add local and global alignment (ablation) | AlignMamba · CMU-MOSI | Accuracy: 82.5 -> 86.9 | Helped |
| Add local and global alignment (ablation) | AlignMamba · CMU-MOSEI | Accuracy: 82.8 -> 86.6 | Helped |
| Use AlignMamba fusion instead of single-stream Mamba (ablation) | Mamba multimodal fusion · CMU-MOSI | Accuracy: 82.3 -> 86.9 | Helped |
| Use AlignMamba fusion instead of multi-stream Mamba (ablation) | Mamba multimodal fusion · CMU-MOSEI | Accuracy: 83.5 -> 86.6 | Helped |
| Remove audio modality (ablation) | AlignMamba · CMU-MOSI | Accuracy: 86.9 -> 84.4 | Hurt |
| Remove video modality (ablation) | AlignMamba · CMU-MOSI | Accuracy: 86.9 -> 83.7 | Hurt |
| Remove language modality (ablation) | AlignMamba · CMU-MOSI | Accuracy: 86.9 -> 65.3 | Hurt |
| Use AlignMamba instead of the strongest reported baseline | Complete multimodal fusion · CMU-MOSI | Accuracy: 86.0 -> 86.9 | Helped |
Surprising findings
- Removing the text modality causes a far larger drop (to 65.3% on CMU-MOSI) than removing audio or video, suggesting language dominates sentiment signal.
Limitations
- Efficiency comparison covers only the fusion component, excluding unimodal encoders.
- Evaluated only on CMU-MOSI and CMU-MOSEI sentiment datasets.
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI
Benchmarks: CMU-MOSI CMU-MOSEI
Metrics: binary accuracy binary F1 GPU memory inference time FLOPs A-distance
Baselines: MulT Self-MM MMIM HyCon ConFEDE AOBERT DMD MTMD IMDer GCNet MMIN MCTN
Builds on: Mamba Optimal transport (relaxed formulation following prior work) Maximum Mean Discrepancy
Closest papers by meaning
- EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (ICLR 2025 · similarity 0.90)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.89)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.89)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.89)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.