Research map / Vision-language and multimodal / Cross-modal learning
Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input
Qingpei Guo, Kaisheng Yao, Wei Chu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Switch-Input and Switch-Attention blocks to plain single-stream BERT (ablation) | BERT encoder · VQAv2 test-dev | VQA-score: 67.8 -> 70.7 | Improved |
| add Switch-Input and Switch-Attention blocks to plain single-stream BERT (ablation) | BERT encoder · RefCOCO+ | accuracy: 69.5 -> 72.8 | Improved |
| add Switch-Attention Block alone to plain BERT (ablation) | BERT encoder · VQAv2 test-dev | VQA-score: 67.8 -> 68.9 | Improved |
| add Switch-Input Block to SAB-only model (ablation) | Switch-BERT · VQAv2 test-dev | VQA-score: 68.9 -> 70.7 | Improved |
| add Switch-Input Block to SAB-only model (ablation) | Switch-BERT · RefCOCO+ | accuracy: 72.4 -> 72.8 | Improved |
| add Switch-Attention Block to SIB-only model (ablation) | Switch-BERT · Flickr30k text retrieval | Recall@1: 77.7 -> 78.2 | Improved |
| add Joint-Attention mode to Cross-Self and Self-Cross modes (pretrained) (ablation) | Switch-Attention Block · RefCOCO+ | accuracy: 71.5 -> 72.8 | Improved |
| add Joint-Attention mode to Cross-Self and Self-Cross modes (no pretraining) (ablation) | Switch-Attention Block · RefCOCO+ | accuracy: 68.3 -> 68.9 | Improved |
| add Cross-Self and Self-Cross modes to Joint-Attention (pretrained) (ablation) | Switch-Attention Block · RefCOCO+ | accuracy: 71.2 -> 72.8 | Improved |
| add pretraining on Conceptual Captions (ablation) | Switch-BERT · Flickr30k image retrieval | Recall@1: 38.2 -> 62.2 | Improved |
| add pretraining on Conceptual Captions (ablation) | Switch-BERT · VQAv2 test-dev | VQA-score: 66.7 -> 70.7 | Improved |
| increase depth from 6 to 12 layers (ablation) | Switch-BERT · VQAv2 test-dev | VQA-score: 65.4 -> 70.7 | Improved |
Closest papers by meaning
- Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.86)
- ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.86)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.85)
- Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · similarity 0.85)
- Wings: Learning Multimodal LLMs without Text-only Forgetting (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
