Research map / Vision-language and multimodal / Cross-modal learning
MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MODA, a modular duplex attention mechanism (Duplex V/T-Aligner via Gram-matrix bases plus modular masked attention) that fixes cross-modal attention imbalance in MLLMs, evaluated on 21 perception, cognition, and emotion benchmarks.
Problem
MLLMs show inconsistent cross-modal attention across layers (attention deficit disorder), biased toward language and decaying with depth, which hurts fine-grained perception, cognition, and emotion understanding.
Approach
Splits multimodal attention into self-modal and cross-modal parts with modality-specific masks, and aligns tokens across modalities by mapping them into duplex modality spaces using bases from normalized Gram matrices (correct-after-align), with an adaptive masked attention that adds decaying pseudo-scores.
Key results
- MODA-34B averages 76.7 on general perception vs. 76.8 for Cambrian-1-34B, with higher knowledge (69.5 vs 67.0) and OCR and chart average (74.7 vs 71.9 for Cambrian-34B-level comparison in table).
- MODA-8B reaches 0.972 on the MMRole cognition average vs. 0.968 for LLaVA-NeXT-8B and 0.895 for Cambrian-1-8B (reported range 0.979-0.981 in text for open-ended models).
- MODA-8B achieves 0.841 average on emotion benchmarks, above the emotion-specialized SPFVTE and MULSER on the reported datasets (e.g., HFM ACC 0.885 vs 0.883).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DAA to MDM (ablation) | MODA-8B · Vision-centric perception (V) | score: 42.6 -> 54.3 | Helped |
| Add MDM to DAA (ablation) | MODA-8B · OCR & Chart perception (O) | score: 63.3 -> 67.0 | Helped |
| Use covariance-based alignment (+CoV) instead of MLP alignment (ablation) | MODA-8B · Vision-centric perception (V) | score: 46.0 -> 54.3 | Helped |
| Use +2xMLP alignment instead of +CoV (ablation) | MODA-8B · General perception (G) | score: 69.3 -> 66.5 | Hurt |
| Use +GeLU alignment instead of +CoV (ablation) | MODA-8B · OCR & Chart perception (O) | score: 67.0 -> 64.0 | Hurt |
| Use Con fusion instead of Add fusion (ablation) | MODA-8B · General perception (G) | score: 62.2 -> 69.3 | Helped |
| Use Fix mask instead of Inf mask (ablation) | MODA-8B · Vision-centric perception (V) | score: 48.1 -> 52.3 | Helped |
| Use adaptive Attn. mask instead of Fix mask (ablation) | MODA-8B · Vision-centric perception (V) | score: 52.3 -> 54.3 | Helped |
| Use MODA-8B instead of Cambrian-1-8B | perception models with Llama-3-Ins-8B · Vision-centric perception average | score: 65.0 -> 66.0 | Helped |
| Use MODA-34B instead of Cambrian-1-34B | perception models with Hermes2-Yi-34B · Knowledge perception average | score: 67.0 -> 69.5 | Helped |
| Use MODA-34B instead of Cambrian-1-34B | perception models with Hermes2-Yi-34B · General perception average | score: 76.8 -> 76.7 | Hurt |
| Use MODA-8B instead of LLaVA-NeXT-8B | cognition models · MMRole cognition average | score: 0.968 -> 0.972 | Helped |
Surprising findings
- Advanced MLLMs reportedly perform near random guessing on 2-class sarcasm detection (HFM), around 50:50 accuracy for three SOTA models.
Limitations
- Ablation and attention analysis details are sparse; the mechanism's formulation (e.g., how the Gram-based fuser is trained via LoRA) is described only briefly.
- Emotion and cognition results rely on GPT-4-based or dataset-specific scoring, and the paper reports mixed wins versus specialized baselines (e.g., Mini-Gemini wins on MVSA-M F1).
Experimental setup
Datasets: MME MMBench SEED GQA ScienceQA MMMU MathVista AI2D ChartQA OCRBench TextVQA DocVQA
Benchmarks: MME MMBench SEED GQA ScienceQA MMMU MathVista AI2D ChartQA OCRBench TextVQA DocVQA
Metrics: accuracy F1 GPT-4 score-based response rating
Baselines: Mini-Gemini-HD LLaVA-NeXT Cambrian-1 MMRole-9B M2CL MULSER CMGCN SPFVTE
Builds on: LLaVA-NeXT Cambrian-1 Mini-Gemini
Closest papers by meaning
- Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.88)
- Adaptive Multimodal Fusion: Dynamic Attention Allocation for Intent Recognition (AAAI 2025 · similarity 0.88)
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.88)
- Multi-to-Single: Reducing Multimodal Dependency in Emotion Recognition Through Contrastive Learning (AAAI 2025 · similarity 0.87)
- Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.