Research map / Vision-language and multimodal / Cross-modal learning
MutualFormer: Multi-modal Representation Learning via Cross-Diffusion Attention
Xixi Wang, Xiao Wang, Bo Jiang, Jin Tang, Bin Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CAM-based adaptive fusion in HAMNet with MutualFormer fusion | modified HAMNet · RGBN300 | mAP: 64.9 -> 69.5 | Improved |
| replace feature-based cross-attention with cross-diffusion attention (ablation) | CrossFormer · NLPR | F-measure (max): 0.917 -> 0.923 | Improved |
| use MutualFormer (self-attention plus CDA) instead of plain Transformer fusion (ablation) | RGB-D SOD fusion module · NJU2K | S-measure: 0.917 -> 0.922 | Improved |
| use Transformer fusion instead of element-wise addition (ablation) | RGB-D SOD fusion module · NLPR | F-measure (max): 0.908 -> 0.923 | Improved |
| use concatenation instead of addition for fusion (ablation) | RGB-D SOD fusion module · NLPR / DUT-RGBD | F-measure (max): 0.908 -> 0.912 on NLPR, 0.932 -> 0.922 on DUT-RGBD | Mixed |
| replace self-attention with cross-attention in Transformer fusion (ablation) | Transformer fusion · NLPR | F-measure (max): 0.923 -> 0.917 | Hurt |
| increase MutualFormer layers from 1 to 2 per level (ablation) | MutualFormer SOD · LFSD | S-measure: 0.865 -> 0.872 | Improved |
| increase MutualFormer layers from 2 to 3 (ablation) | MutualFormer SOD · LFSD | F-measure (max): 0.879 -> 0.864 | Hurt |
| set diffusion balance epsilon to 0.6 instead of 0.0 (ablation) | MutualFormer CDA · LFSD | S-measure: 0.862 -> 0.872 | Improved |
| set diffusion balance epsilon to 0.6 instead of 0.7 (ablation) | MutualFormer CDA · LFSD | S-measure: 0.861 -> 0.872 | Improved |
| add focal regularization weight lambda 0.4 versus 0.0 (ablation) | MutualFormer SOD · LFSD | F-measure (max): 0.871 -> 0.879 | Improved |
| add focal regularization weight lambda 0.4 versus 0.0 (ablation) | MutualFormer SOD · NJU2K | S-measure: 0.920 -> 0.922 | No clear effect |
Closest papers by meaning
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.86)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.85)
- Diffusion Models for Multi-Task Generative Modeling (ICLR 2024 · similarity 0.85)
- Learning Shared Representations from Unpaired Data (NeurIPS 2025 · similarity 0.84)
- Cross-modal Representation Flattening for Multi-modal Domain Generalization (NeurIPS 2024 · similarity 0.84)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
