AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

MutualFormer: Multi-modal Representation Learning via Cross-Diffusion Attention

Xixi Wang, Xiao Wang, Bo Jiang, Jin Tang, Bin Luo

IJCV 2024 · Vision-language and multimodal › Cross-modal learning · approach: Multi-modality image fusion · 34 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace CAM-based adaptive fusion in HAMNet with MutualFormer fusionmodified HAMNet · RGBN300mAP: 64.9 -> 69.5Improved
replace feature-based cross-attention with cross-diffusion attention (ablation)CrossFormer · NLPRF-measure (max): 0.917 -> 0.923Improved
use MutualFormer (self-attention plus CDA) instead of plain Transformer fusion (ablation)RGB-D SOD fusion module · NJU2KS-measure: 0.917 -> 0.922Improved
use Transformer fusion instead of element-wise addition (ablation)RGB-D SOD fusion module · NLPRF-measure (max): 0.908 -> 0.923Improved
use concatenation instead of addition for fusion (ablation)RGB-D SOD fusion module · NLPR / DUT-RGBDF-measure (max): 0.908 -> 0.912 on NLPR, 0.932 -> 0.922 on DUT-RGBDMixed
replace self-attention with cross-attention in Transformer fusion (ablation)Transformer fusion · NLPRF-measure (max): 0.923 -> 0.917Hurt
increase MutualFormer layers from 1 to 2 per level (ablation)MutualFormer SOD · LFSDS-measure: 0.865 -> 0.872Improved
increase MutualFormer layers from 2 to 3 (ablation)MutualFormer SOD · LFSDF-measure (max): 0.879 -> 0.864Hurt
set diffusion balance epsilon to 0.6 instead of 0.0 (ablation)MutualFormer CDA · LFSDS-measure: 0.862 -> 0.872Improved
set diffusion balance epsilon to 0.6 instead of 0.7 (ablation)MutualFormer CDA · LFSDS-measure: 0.861 -> 0.872Improved
add focal regularization weight lambda 0.4 versus 0.0 (ablation)MutualFormer SOD · LFSDF-measure (max): 0.871 -> 0.879Improved
add focal regularization weight lambda 0.4 versus 0.0 (ablation)MutualFormer SOD · NJU2KS-measure: 0.920 -> 0.922No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy