Research map / Vision-language and multimodal / CLIP and zero-shot
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on noisy web alt-text pairs at billion-example scale with a dual-encoder contrastive objective | ALIGN · Flickr30K and MSCOCO zero-shot retrieval | R@1: Flickr30K image-to-text: 88.0 -> 88.6; Flickr30K text-to-image: 68.7 -> 75.7; MSCOCO image-to-text: 58.4 -> 58.6; MSCOCO | Helped |
| fine-tune the ALIGN dual encoder on task-specific retrieval data | ALIGN · Flickr30K and MSCOCO fine-tuned retrieval | R@1: Flickr30K image-to-text: 88.7 -> 95.3; MSCOCO image-to-text: 73.5 -> 77.0 | Helped |
| use cross-modal image-text training rather than intra-modal matching objectives | ALIGN · CxC | R@1: image-to-text 55.9 -> 78.1; text-to-image 41.7 -> 61.8; text-to-text 42.4 -> 45.4 | Mixed |
| use prompt ensembling for class-name text embeddings | ALIGN zero-shot classifier · ImageNet | top-1 accuracy: +2.9% | Helped |
| increase embedding dimension from the 640-dimensional baseline to 1376 for the larger model (ablation) | ALIGN · MSCOCO zero-shot retrieval and ImageNet KNN | R@1 | Helped |
| reduce embedding dimension from 640 to 320 (ablation) | B5 + BERT-base · MSCOCO zero-shot image-to-text | R@1: 51.7 -> 50.3 | Hurt |
| reduce embedding dimension from 640 to 160 (ablation) | B5 + BERT-base · MSCOCO zero-shot image-to-text | R@1: 51.7 -> 47.0 | Hurt |
| reduce embedding dimension from 640 to 80 (ablation) | B5 + BERT-base · MSCOCO zero-shot image-to-text | R@1: 51.7 -> 42.0 | Hurt |
| use only 50% of the in-batch negatives (ablation) | B5 + BERT-base · MSCOCO zero-shot image-to-text | R@1: 51.7 -> 50.2 | Hurt |
| use only 25% of the in-batch negatives (ablation) | B5 + BERT-base · MSCOCO zero-shot image-to-text | R@1: 51.7 -> 48.7 | Hurt |
| fix softmax temperature at 1/128 instead of learning it (ablation) | B5 + BERT-base · MSCOCO zero-shot image-to-text | R@1: 51.7 -> 52.2 | Helped |
| fix softmax temperature at 1/64 instead of learning it (ablation) | B5 + BERT-base · MSCOCO zero-shot image-to-text | R@1: 51.7 -> 52.2 | Helped |
Closest papers by meaning
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.91)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.90)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.90)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.89)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.