Research map / Vision-language and multimodal / Cross-modal learning
MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MANGO, a normalizing flow whose coupling layers are replaced by invertible cross-attention (MMCA, IMCA, LICA partitioning) operating in an autoencoder latent space, evaluated on RGB-D segmentation, multimodal image translation, and MM-IMDB genre classification.
Problem
Implicit attention-based multimodal fusion does not explicitly model cross-modality correlations, while normalizing flows have limited expressiveness for long-range dependencies and scale poorly to high-dimensional multimodal data.
Approach
An invertible cross-attention layer inside a normalizing flow: attention from one token partition to another, made invertible via an autoregressive (triangular) mask, with three partitioning schemes (modality-to-modality, inter-modality, learnable permutation) and a latent autoencoder space to scale the flow.
Key results
- On NYUDv2 MANGO reaches 59.2 mIoU and 81.5 pixel accuracy, versus GeminiFusion (MiT-B5) at 57.7 mIoU.
- On MM-IMDB MANGO reaches 71.7 micro F1 and 68.2 macro F1, versus BridgeTow at 68.2 and 63.3.
- Ablation: adding LICA to MMCA+IMCA raises NYUDv2 mIoU from 58.0 to 59.2; 16 blocks give 61.7 mIoU on NYUDv2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LICA to MMCA + IMCA (ablation) | MANGO · NYUDv2 | mIoU: 58.0 -> 59.2 | Helped |
| Add LICA to MMCA + IMCA (ablation) | MANGO · SUN RGB-D | mIoU: 53.7 -> 54.1 | Helped |
| Add latent model (ablation) | MANGO with 12 blocks · NYUDv2 | mIoU: 58.0 -> 59.2 | Helped |
| Add latent model (ablation) | MANGO with 12 blocks · SUN RGB-D | mIoU: 53.5 -> 54.1 | Helped |
| Increase cross-attention blocks from 6 to 12 (ablation) | MANGO · NYUDv2 | mIoU: 52.3 -> 59.2 | Helped |
| Increase cross-attention blocks from 6 to 12 (ablation) | MANGO · SUN RGB-D | mIoU: 48.1 -> 54.1 | Helped |
| Increase cross-attention blocks from 12 to 16 (ablation) | MANGO · NYUDv2 | mIoU: 59.2 -> 61.7 | Helped |
| Increase cross-attention blocks from 12 to 16 (ablation) | MANGO · SUN RGB-D | mIoU: 54.1 -> 55.6 | Helped |
| Replace coupling layer with MANGO invertible cross-attention (ablation) | Flow model · NYUDv2 | mIoU: 50.8 -> 59.2 | Helped |
| Replace coupling layer with MANGO invertible cross-attention (ablation) | Flow model · SUN RGB-D | mIoU: 48.5 -> 54.1 | Helped |
Limitations
- Loss-weight balancing among learning objectives not explored
- Experiments limited to standard benchmark scale due to compute
- Generalization to larger-scale data is hypothesized, not shown
Experimental setup
Datasets: NYUDv2 SUN RGB-D Taskonomy MM-IMDB
Benchmarks: NYUDv2 SUN RGB-D Taskonomy MM-IMDB
Metrics: Pixel Accuracy mAcc mIoU FID KID MAE MSE Micro F1 Macro F1
Baselines: CEN TokenFusion GeminiFusion Feature Concat Glow Flow++ AttnFlow Coupling Layer BridgeTow BLIP COCA MFM
Builds on: RealNVP Glow TokenFusion GeminiFusion AttnFlow
Closest papers by meaning
- Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process (NeurIPS 2025 · similarity 0.85)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.85)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.84)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.84)
- Adaptive Multimodal Fusion: Dynamic Attention Allocation for Intent Recognition (AAAI 2025 · similarity 0.84)
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.