Research map / Vision-language and multimodal / CLIP and zero-shot
Learning Visual N-Grams From Web Data
Ang Li, Allan Jabri, Armand Joulin, Laurens van der Maaten
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Jelinek-Mercer loss instead of naive n-gram loss, stupid back-off at test (ablation) | visual n-gram model · YFCC100M test | perplexity: 297 -> 276 | Improved |
| use Jelinek-Mercer loss instead of naive n-gram loss, J-M smoothing at test (ablation) | visual n-gram model · YFCC100M test | perplexity: 212 -> 199 | Improved |
| add Jelinek-Mercer smoothing at prediction time instead of stupid back-off (ablation) | Jelinek-Mercer loss model · YFCC100M test | perplexity: 276 -> 199 | Improved |
| use Jelinek-Mercer loss instead of naive n-gram loss (ablation) | visual n-gram model · YFCC100M test | phrase recall@1: 5.5 -> 6.2 | Improved |
| use Jelinek-Mercer loss instead of naive n-gram loss (ablation) | visual n-gram model · YFCC100M test | phrase accuracy (human raters): 36.4 -> 42.0 | Improved |
| train the full network with n-gram loss instead of linear layer on Imagenet features | Imagenet + linear baseline · YFCC100M test | phrase recall@1: 5.0 -> 6.2 | Improved |
| include longer n-grams (up to five) in the dictionary (ablation) | Jelinek-Mercer model · YFCC100M test | recall@k of n-gram retrieval | Improved |
| use Jelinek-Mercer loss instead of naive n-gram loss (ablation) | visual n-gram model · YFCC100M test | caption retrieval accuracy (human raters): 42.0 -> 53.1 | Improved |
| use n-gram dictionary models instead of unigram model (ablation) | Jelinek-Mercer model · YFCC100M test | caption retrieval recall@1: 1.2 -> 7.1 | Improved |
| finetune Jelinek-Mercer model on COCO | Jelinek-Mercer model · COCO-5K image retrieval | recall@1: 5.0 -> 11.0 | Improved |
| finetune Jelinek-Mercer model on COCO | Jelinek-Mercer model · COCO-5K caption retrieval | recall@1: 8.7 -> 17.8 | Improved |
Closest papers by meaning
- Learning Transferable Visual Models From Natural Language Supervision (ICML 2021 · similarity 0.87)
- Learning Deep Representations of Fine-Grained Visual Descriptions (CVPR 2016 · similarity 0.87)
- Revisiting Image-Language Networks for Open-Ended Phrase Detection (TPAMI 2020 · similarity 0.87)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.87)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.86)
- Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
