Research map / Vision-language and multimodal / CLIP and zero-shot
Linearly Mapping from Image to Text Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP with natural-language caption pretraining rather than NFRN50 with image-classification pretraining | LiMBeR · NoCaps (All) | CIDEr-D: 28.5 -> 43.9 | Helped |
| Use image-classification-finetuned BEIT instead of pretrained BEIT (ablation) | LiMBeR · NoCaps (All) | CIDEr-D: 18.9 -> 45.3 | Helped |
| Use pretrained BEIT rather than randomly initialized NFRN50 (ablation) | LiMBeR · NoCaps (All) | CIDEr-D: 5.0 -> 18.9 | Helped |
| Tune the image encoder and language model in MAGMA rather than training only the projection with frozen models | LiMBeR CLIP · COCO | CIDEr-D: 54.9 -> 47.5 | Hurt |
| Use pretrained BEIT rather than randomly initialized BEIT (ablation) | LiMBeR · COCO 2017 val | CIDEr: 0.052 -> 0.223 | Helped |
| Use BEIT rather than the blind LM with no image input | LiMBeR · VQA2 (4-shot) | accuracy: 36.99 -> 31.72 | Hurt |
| Use CLIP rather than BEIT as the image encoder | LiMBeR · VQA2 (0-shot) | accuracy: 24.92 -> 33.33 | Helped |
| Add image-classification finetuning to BEIT (ablation) | LiMBeR · COCO | CLIPScore: 63.6 -> 74.2 | Helped |
| Use a linguistically supervised encoder rather than vision-only BEIT | LiMBeR · AWA2 | animal-name accuracy: 13% -> 43% (NFRN50); 59% (CLIP) | Helped |
| Use BEIT rather than NFRN50 for transfer of perceptual similarity in mistaken animal labels | LiMBeR · AWA2 | Wu-Palmer similarity: BEIT: 0.8, NFRN50: 0.81, CLIP: 0.8 | No clear effect |
Closest papers by meaning
- A Vision Check-up for Language Models (CVPR 2024 · similarity 0.91)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.91)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.90)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.90)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.