Research map / Vision-language and multimodal / Cross-modal learning
Multimodal Token Fusion for Vision Transformers
Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, Yunhe Wang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace less informative tokens with aligned features from other modalities, using TokenFusion | Vision transformers · Taskonomy Shade+Texture→RGB | FID: 62.63 -> 43.92 | Improved |
| Replace less informative tokens with aligned features from other modalities, using TokenFusion | Vision transformers · NYUDv2 RGB-depth segmentation | mIoU: 52.5 -> 54.2 | Improved |
| Replace less informative point tokens with aligned image features | Group-Free 3D detector · SUN RGB-D 3D detection | mAP@0.25: 63.0 -> 64.9 | Improved |
| Add L1 sparsity and learned token fusion (ablation) | RGB-depth segmentation · NYUDv2 | mIoU: 49.7 -> 53.3 | Improved |
| Add L1 sparsity and learned token fusion (ablation) | 3D detection with images and points · SUN RGB-D | mAP@0.25: 62.8 -> 64.9 | Improved |
| Replace learned token selection with random 10% fusion (ablation) | RGB-depth segmentation · NYUDv2 | mIoU: 49.7 -> 50.1 | No clear effect |
| Replace learned token selection with random 30% fusion (ablation) | RGB-depth segmentation · NYUDv2 | mIoU: 49.7 -> 48.2 | Hurt |
| Add RPA without token fusion (ablation) | RGB-depth segmentation · NYUDv2 | mIoU: 49.7 -> 50.3 | No clear effect |
| Add RPA to token fusion (ablation) | 3D detection with images and points · SUN RGB-D | mAP@0.5: 46.2 -> 48.3 | Improved |
| Share MSA and MLP across modalities while keeping layer normalization modality-specific (ablation) | TokenFusion · Taskonomy Shade+Texture→RGB | FID: 49.73 -> 43.92 | Improved |
| Share layer normalization across modalities (ablation) | TokenFusion with shared MSA and MLP · Taskonomy Shade+Texture→RGB | FID: 43.92 -> 67.45 | Hurt |
| Add channel-wise pruning to token-wise TokenFusion (ablation) | RGB-depth segmentation · NYUDv2 | mIoU: 53.3 -> 53.8 | Improved |
Closest papers by meaning
- GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer (ICML 2024 · similarity 0.91)
- Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers (CVPR 2024 · similarity 0.88)
- MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer (CVPR 2024 · similarity 0.88)
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (ICLR 2026 · similarity 0.87)
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.87)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
