Research map / Vision-language and multimodal / Human and embodied modeling
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ITC before multimodal fusion (ablation) | ALBEF pre-training · Downstream V+L tasks | TR average recall on Flickr30K: 93.96 -> 96.55 | Helped |
| Add contrastive hard negative mining (ablation) | ALBEF pre-training · Flickr30K fine-tuned test | TR average recall: 96.55 -> 97.01 | Helped |
| Add momentum distillation to ITC (ablation) | ALBEF pre-training · VQA test-dev | score: 73.81 -> 74.06 | Helped |
| Add momentum distillation to MLM (ablation) | ALBEF pre-training · VQA test-dev | score: 74.06 -> 74.42 | Helped |
| Apply momentum distillation during downstream fine-tuning (ablation) | Full ALBEF · Flickr30K fine-tuned test | TR average recall: 97.47 -> 97.83 | Helped |
| Increase pre-training data from 4M to 14M images | ALBEF · Flickr30K zero-shot test | TR R@1: 90.5 -> 94.1 | Helped |
| Remove hard negative mining (ablation) | ALBEF retrieval fine-tuning · Flickr30K test | TR average recall: 98.57 -> 98.22 (-0.35) | Hurt |
| Vary candidate count k for ITM reranking (ablation) | ALBEF retrieval inference · Flickr30K fine-tuned test | TR average recall: 98.60; 98.57; 98.57 | No clear effect |
| Add text-assignment pre-training (ablation) | NLVR² model with shared cross-attention · NLVR² test-P | accuracy: 80.45 -> 83.14 | Helped |
| Share only cross-attention parameters instead of sharing all block parameters (ablation) | NLVR² model with TA pre-training · NLVR² test-P | accuracy: 82.36 -> 83.14 | Helped |
Closest papers by meaning
- Expediting Contrastive Language-Image Pretraining via Self-Distilled Encoders (AAAI 2024 · similarity 0.90)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.89)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.89)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.