Research map / Vision-language and multimodal / Cross-modal learning
EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
EMMA adds a pixel-wise alignment loss (decoding visual tokens back to the image) and a multi-scale feature fusion module over intermediate Mamba LLM layers to improve visual fidelity and cross-modal alignment in Cobra-style Mamba MLLMs.
Problem
Mamba-based multimodal LLMs extract visual features poorly inside the LLM, and visual latents degrade across layers, causing imbalanced cross-modal alignment, weaker multimodal performance, and more hallucination.
Approach
Extend next-token prediction to visual tokens and supervise the reconstructed image with an L2 pixel-wise loss, plus fuse intermediate-layer visual features via cross-attention and Mamba blocks so that fine-grained visual information is preserved throughout the LLM.
Key results
- EMMA-V1 reaches 1572.8 on MME versus 1294.3 for Cobra, and 52.1 VizWiz / 57.2 TextVQA versus Cobra's 52.0 / 52.4.
- EMMA-V1 scores 88.0 on POPE and 51.0 on HallusionBench versus 87.2 and 41.4 for Cobra.
- EMMA-V2 (MambaV2-2.7B) decodes 149.96 tokens/s versus 138.95 for Cobra and about 41 for TinyLLaVA and 39 for MobileVLM V2 on a single A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add structural and hierarchical alignment with EMMA | Cobra / MambaV1-2.8B · MME | score: 1294.3 -> 1572.8 | Helped |
| add structural and hierarchical alignment with EMMA | Cobra / MambaV1-2.8B · TextVQA | score: 52.4 -> 57.2 | Helped |
| add structural and hierarchical alignment with EMMA | Cobra / MambaV1-2.8B · VQAv2 | score: 74.9 -> 76.3 | Helped |
| add structural and hierarchical alignment with EMMA | Cobra / MambaV1-2.8B · VSR | score: 51.7 -> 51.5 | Hurt |
| use EMMA-V2 rather than Cobra | Cobra · VizWiz | score: 52.0 -> 54.1 | Helped |
| use EMMA-V1 rather than Cobra | Cobra · HallusionBench | score: 41.4 -> 51.0 | Helped |
| use EMMA rather than Cobra | Cobra · tokens generated per second | tokens/second: 138.95 -> 149.96 | Helped |
| add multi-scale feature fusion (MFF) (ablation) | EMMA · MME | score: 1294.1 -> 1572.8 | Helped |
| add multi-scale feature fusion (MFF) (ablation) | EMMA · GQA | score: 59.3 -> 60.5 | Helped |
| add pixel-wise alignment loss (PAL) (ablation) | EMMA · TextVQA | score: 52.4 -> 57.2 | Helped |
| add pixel-wise alignment loss (PAL) (ablation) | EMMA · HallusionBench | score: 41.4 -> 51.0 | Helped |
| add cross-scan mechanism (CSM) (ablation) | EMMA · VizWiz | score: 52.1 -> 54.2 | Helped |
Surprising findings
- Aligning intermediate encoder features instead of raw image pixels (+AVF) caused a large drop across metrics (e.g., MME 984.8, MMB 25.0), though HallusionBench stayed relatively high at 47.2.
- Adding a visual selective scan cross-scan module on top of the method gave only marginal changes, suggesting the structural constraint may already suffice.
Limitations
- Results on VQAv2 trail VL-Mamba slightly, and many scores are about 1 point below the best transformer model.
- Latency for VL-Mamba is assumed from Cobra rather than measured, since it is not open-sourced.
- Code is promised ('Code will be provided') but not yet released; gains over Cobra are evaluated mainly on a single 2.8B backbone.
Experimental setup
Datasets: LLaVA-v1.5-mixed-665k LVIS-Instruct-4V LRV-Instruct VQAv2 GQA VizWiz TextVQA VSR POPE HallusionBench MMBench MME
Benchmarks: VQAv2 GQA VizWiz TextVQA VSR MME MMBench SEED-Bench ScienceQA AI2D CCBench POPE
Metrics: accuracy MME perception+cognition score tokens per second average time per 256 tokens
Baselines: Cobra VL-Mamba TinyLLaVA MobileVLM V2 LLaVA-1.5 EMU2 MoE-LLaVA Qwen-VL
Builds on: Cobra Mamba Mamba2 LLaVA
Closest papers by meaning
- AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.90)
- Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation (CVPR 2025 · similarity 0.86)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.86)
- Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference (AAAI 2025 · similarity 0.86)
- DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.86)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.