Research map / Vision-language and multimodal / Cross-modal learning
Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision
Andrew Shin, Masato Ishii, Takuya Narihira
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add large-scale pre-training (ablation) | ViLBERT · VQA (dev) | accuracy: 68.93 -> 70.55 | Improved |
| add large-scale pre-training (ablation) | VL-BERT · VQA (dev) | accuracy: 69.58 -> 71.16 | Improved |
| add pre-training (ablation) | InterBERT · VCR (Q/A) | accuracy: 63.6 -> 73.1 | Improved |
| add pre-training (ablation) | Unicoder-VL · Image retrieval | R@1: 57.8 -> 71.5 | Improved |
| add pre-training (ablation) | ViLBERT · Image retrieval | R@1: 45.50 -> 58.20 | Improved |
| scale model from base to large | VL-BERT · VQA (dev) | accuracy: 71.16 -> 71.79 | Improved |
| scale model from base to large | UNITER · VQA (dev) | accuracy: 72.70 -> 73.82 | Improved |
| scale image backbone from ResNet-50 to ResNeXt-152 | PixelBERT · VQA (dev) | accuracy: 71.35 -> 74.45 | Improved |
| add Visual Genome to COCO pre-training data | LAMP · VQA (dev) | accuracy: 70.85 -> 72.48 | Improved |
| pre-train on video-text with YouCook2 annotations (UniVL) instead of YouTube-only (VideoBERT) | video-language transformers · YouCook2 | CIDEr: 0.55 -> 1.81 | Improved |
| add dataset-matched pre-training for target task (image QA data) | LXMERT, OSCAR, SemVLP · VQA (dev) | accuracy | Improved |
Closest papers by meaning
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (ICML 2025 · similarity 0.88)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.87)
- Circuit Component Reuse Across Tasks in Transformer Language Models (ICLR 2024 · similarity 0.86)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.86)
- Do Transformer Interpretability Methods Transfer to RNNs? (AAAI 2025 · similarity 0.86)
- Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
