Research map / Vision-language and multimodal / Multimodal LLMs
CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
Moritz Böhle, Amelie Royer, Juliette Marrie, Edouard Grave, Patrick Perez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace cross-attention with token insertion | Helium1-2B VLM · ChartQA | accuracy: 75.9 -> 80.0 | Improved |
| replace cross-attention with token insertion | Helium1-2B VLM · InfoVQA | accuracy: 51.8 -> 55.3 | Improved |
| use cross-attention instead of token insertion | Helium1-2B VLM · MME | score: 1676 -> 1731 | Improved |
| use cross-attention instead of token insertion | Helium1-2B VLM · streaming inference | FPS: 1.2 -> 6.8 | Improved |
| use cross-attention instead of token insertion | Helium1-2B VLM · streaming inference | memory: 29.5 -> 6.1 | Improved |
| add text-to-text attention in CA layers (CA t+v) (ablation) | CA (He-2B) · RealWorldQA | accuracy: 58.0 -> 60.3 | Improved |
| add text-to-text attention in CA layers (CA t+v) (ablation) | CA (He-2B) · MME | score: 1731 -> 1720 | No clear effect |
| share projection parameters between CA and SA (ablation) | CA (He-2B) · ChartQA | accuracy: 75.9 -> 74.1 | Hurt |
| share projection parameters between CA and SA (ablation) | CA (He-2B) · MME | score: 1731 -> 1613 | Hurt |
| share projection parameters between CA and SA (ablation) | CA (He-2B) · streaming inference | FPS: 6.8 -> 7.7 | Improved |
| replace every other SA layer with CA (ablation) | CA (He-2B) · ChartQA | accuracy: 75.9 -> 73.8 | Hurt |
| replace every other SA layer with CA (ablation) | CA (He-2B) · training | memory: 32.9 -> 29.0 | Improved |
Closest papers by meaning
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.89)
- LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models (ICML 2025 · similarity 0.89)
- CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers (ICML 2024 · similarity 0.89)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.88)
- COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training (CVPR 2025 · similarity 0.88)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
