AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Multimodal Token Fusion for Vision Transformers

Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, Yunhe Wang

CVPR 2022 · Vision-language and multimodal › Cross-modal learning · approach: Multi-modality image fusion · 246 citations (OpenAlex)

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace less informative tokens with aligned features from other modalities, using TokenFusionVision transformers · Taskonomy Shade+Texture→RGBFID: 62.63 -> 43.92Improved
Replace less informative tokens with aligned features from other modalities, using TokenFusionVision transformers · NYUDv2 RGB-depth segmentationmIoU: 52.5 -> 54.2Improved
Replace less informative point tokens with aligned image featuresGroup-Free 3D detector · SUN RGB-D 3D detectionmAP@0.25: 63.0 -> 64.9Improved
Add L1 sparsity and learned token fusion (ablation)RGB-depth segmentation · NYUDv2mIoU: 49.7 -> 53.3Improved
Add L1 sparsity and learned token fusion (ablation)3D detection with images and points · SUN RGB-DmAP@0.25: 62.8 -> 64.9Improved
Replace learned token selection with random 10% fusion (ablation)RGB-depth segmentation · NYUDv2mIoU: 49.7 -> 50.1No clear effect
Replace learned token selection with random 30% fusion (ablation)RGB-depth segmentation · NYUDv2mIoU: 49.7 -> 48.2Hurt
Add RPA without token fusion (ablation)RGB-depth segmentation · NYUDv2mIoU: 49.7 -> 50.3No clear effect
Add RPA to token fusion (ablation)3D detection with images and points · SUN RGB-DmAP@0.5: 46.2 -> 48.3Improved
Share MSA and MLP across modalities while keeping layer normalization modality-specific (ablation)TokenFusion · Taskonomy Shade+Texture→RGBFID: 49.73 -> 43.92Improved
Share layer normalization across modalities (ablation)TokenFusion with shared MSA and MLP · Taskonomy Shade+Texture→RGBFID: 43.92 -> 67.45Hurt
Add channel-wise pruning to token-wise TokenFusion (ablation)RGB-depth segmentation · NYUDv2mIoU: 53.3 -> 53.8Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy