Research map / Vision-language and multimodal / Human and embodied modeling
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use co-attention instead of merged attention (ablation) | METER · VQAv2 test-dev | accuracy: 74.00 -> 74.98 | Helped |
| Use co-attention instead of merged attention (ablation) | METER · Flickr30k zero-shot | IR@1: 57.46 -> 66.08 | Helped |
| Add a decoder to the co-attention encoder-only architecture (ablation) | METER · Flickr30k zero-shot | IR@1: 66.08 -> 48.96 | Hurt |
| Add masked patch classification with in-batch negatives (ablation) | METER with MLM + ITM · VQAv2 test-dev | accuracy: 74.98 -> 74.01 | Hurt |
| Add masked patch classification with discrete codes (ablation) | METER with MLM + ITM · Flickr30k zero-shot | IR@1: 66.08 -> 59.80 | Hurt |
| Remove ITM from MLM + ITM pre-training (ablation) | METER · VQAv2 test-dev | accuracy: 74.98 -> 74.19 | Hurt |
| Remove MLM from MLM + ITM pre-training (ablation) | METER · VQAv2 test-dev | accuracy: 74.98 -> 72.63 | Hurt |
| Replace CLIP-ViT-224/32 with CLIP-ViT-224/16 (ablation) | METER with RoBERTa · VQAv2 test-dev | accuracy: 74.67 -> 77.19 | Helped |
| Replace the pretrained text encoder with word embeddings only (ablation) | METER with CLIP-ViT-224/32 · VQAv2 test-dev | accuracy: 74.67 -> 73.99 | Hurt |
| Use a larger learning rate for randomly initialized layers than pretrained parameters (ablation) | METER · VQAv2 test-dev | accuracy: 73.16 -> 74.98 | Helped |
| Fuse representations from multiple encoder layers (ablation) | METER-CLIP-ViT_BASE after VLP · VQAv2 test-dev | accuracy: 77.19 -> 77.06 | Hurt |
| Fuse representations from multiple encoder layers (ablation) | METER-Swin_BASE without VLP · VQAv2 test-dev | accuracy: 72.38 -> 72.91 | Helped |
Closest papers by meaning
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.90)
- Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.90)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding (CVPR 2025 · similarity 0.89)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.