Research map / Vision-language and multimodal / Cross-modal learning
GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GeminiFusion, a pixel-wise multimodal fusion module for vision transformers that combines intra-modal self-attention with spatially co-located cross-modal attention, using a relation discriminator and layer-adaptive noise to reach linear complexity and beat exchange-based TokenFusion on segmentation, translation, and 3D detection.
Problem
Cross-attention fusion for multimodal vision transformers has quadratic cost in the number of tokens, while exchange-based fusion (TokenFusion) is cheap but underperforms and can lose informative tokens.
Approach
Restrict cross-modal attention to spatially co-located (same-index) tokens across modalities, making attention per-pixel and linear in token count; add a 1x1 conv + softmax relation discriminator to modulate keys, intra-modal self-attention terms, and learnable per-layer noise added to keys and values to balance self- and cross-modal attention.
Key results
- On NYUDv2 with MiT-B5, GeminiFusion reaches 57.7 mIoU versus 55.1 for reproduced TokenFusion; with Swin-Large-22k it reaches 60.2 mIoU on NYUDv2 and 54.6 on SUN RGB-D.
- On DeLiVER with four modalities (RGB+D+E+L), GeminiFusion with MiT-B2 achieves 66.9 mIoU versus 63.5 for TokenFusion.
- Compared with cross-attention, the pixel-wise module reduces FLOPs from about 17G to 0.14G per attention instance, and the full GeminiFusion model uses 174 GFLOPs versus 108 for TokenFusion on NYUDv2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace TokenFusion with GeminiFusion | MiT-B3 multimodal segmentation · NYUDv2, RGB+D | mIoU: 54.2 -> 56.8 | Helped |
| Replace TokenFusion with GeminiFusion | MiT-B5 multimodal segmentation · SUN RGB-D, RGB+D | mIoU: 51.8 -> 53.3 | Helped |
| Replace TokenFusion with GeminiFusion | MiT-B2 multimodal segmentation · DeLiVER, RGB+D+E+L | mIoU: 63.5 -> 66.9 | Helped |
| Replace TokenFusion with GeminiFusion | Taskonomy image-to-image translation · Shade+Texture→RGB | FID: 47.31 -> 41.32 | Helped |
| Add GeminiFusion to the MVX-Net fusion layer | MVX-Net · KITTI validation, 3D AP R11 Easy | 3D AP R11 Easy: 87.49 -> 88.49 | Helped |
| Use 2-layer MLP with Softmax relation discriminator (ablation) | GeminiFusion · NYUDv2 | mIoU: 55.7 -> 56.8 | Helped |
| Replace Softmax with Sigmoid in the relation discriminator (ablation) | GeminiFusion · NYUDv2 | mIoU: 56.8 -> 55.9 | Hurt |
| Use a 1x1 CNN with Softmax instead of the 2-layer MLP with Softmax (ablation) | GeminiFusion · NYUDv2 | mIoU: 56.8 -> 55.7 | Hurt |
| Add a learnable noise parameter instead of multiplying by it (ablation) | GeminiFusion · NYUDv2 | mIoU: 56.2 -> 56.8 | Helped |
| Add a learnable noise parameter instead of random Gaussian noise (ablation) | GeminiFusion · NYUDv2 | mIoU: 55.3 -> 56.8 | Helped |
| Add point-wise cross-attention (ablation) | GeminiFusion · NYUDv2 | mIoU: 53.3 -> 55.4 | Helped |
| Add noise self-attention (ablation) | GeminiFusion · NYUDv2 | mIoU: 55.4 -> 56.3 | Helped |
Surprising findings
- Unconditionally exchanging all tokens (threshold 1) performed better than TokenFusion's learned pruning-based exchange.
- Applying GeminiFusion only in the last layer already beats TokenFusion in both latency and accuracy, while full-depth application gives the best accuracy.
Limitations
- Only handles homogeneous modalities that are spatially aligned as image-like inputs; heterogeneous pairs such as image-audio or image-text are not supported.
- Requires pre-defined alignment between modality pairs.
- Improvements over TokenFusion on some metrics are modest (e.g., 0.2-0.5 points on some settings).
Experimental setup
Datasets: NYUDv2 SUN RGB-D DeLiVER Taskonomy KITTI ImageNet-1K
Benchmarks: NYUDv2 SUN RGB-D DeLiVER Taskonomy KITTI
Metrics: mIoU pixel accuracy mAcc FID KID MAE MSE 3D AP GFLOPs latency
Baselines: TokenFusion CMNeXt CMX DFormer DPLNet MVX-Net OmniVec
Builds on: TokenFusion SegFormer CMNeXt MVX-Net Swin Transformer
Closest papers by meaning
- LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer (ICML 2026 · similarity 0.85)
- Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs (NeurIPS 2026 · similarity 0.85)
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.85)
- PixelLM: Pixel Reasoning with Large Multimodal Model (CVPR 2024 · similarity 0.85)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.85)
- When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.