Research map / Vision-language and multimodal / Cross-modal learning
The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, Zilong Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add bidirectional attention among image patches with multimodal RoPE (ablation) | SAIL-0.5B · TextVQA | accuracy: 16.2 -> 20.1 | Improved |
| add bidirectional attention among image patches with multimodal RoPE (ablation) | SAIL-0.5B · VQAv2 | accuracy: 57.8 -> 59.1 | Improved |
| add bidirectional attention among image patches with multimodal RoPE (ablation) | SAIL-0.5B · GQA | accuracy: 45.2 -> 46.9 | Improved |
| add bidirectional attention among image patches with multimodal RoPE (ablation) | SAIL-0.5B · SEEDBench-IMG | accuracy: 33.8 -> 35.1 | Improved |
| interleave pure text data (SlimPajama) in pretraining (ablation) | SAIL-0.5B · ScienceQA | accuracy: 48.6 -> 59.6 | Improved |
| interleave pure text data (SlimPajama) in pretraining (ablation) | SAIL-0.5B · GQA | accuracy: 42.1 -> 46.9 | Improved |
| interleave pure text data (SlimPajama) in pretraining (ablation) | SAIL-0.5B · TextVQA | accuracy: 18.3 -> 20.1 | Improved |
| scale pretraining data from 32M to 512M pairs (single transformer vs modular MLLM) | SAIL · Normalized average benchmark performance | performance | Improved |
| scale model size from 0.5B to 3B to 7B | SAIL · downstream VLM tasks average | average performance | Improved |
| single transformer (SAIL) vs modular MLLM on spatial reasoning | SAIL · MMBench Physical Relation | accuracy: 30.4 -> 52.2 | Improved |
| single transformer (SAIL) vs modular MLLM on spatial position | SAIL · MME Position | score: 98.3 -> 160.0 | Improved |
| single transformer (SAIL) vs modular MLLM on celebrity knowledge | SAIL · MME Celebrity | score: 134.0 -> 108.2 | Hurt |
Closest papers by meaning
- Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model (NeurIPS 2024 · similarity 0.89)
- Assessing and Learning Alignment of Unimodal Vision and Language Models (CVPR 2025 · similarity 0.89)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.88)
- EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE (AAAI 2024 · similarity 0.88)
- Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
