Research map / Vision-language and multimodal / Human and embodied modeling
EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvdCLIP, which uses LLM-generated entity visual descriptions (EVDs) and a trainable T5-based EVD-aware rewriter (EaRW) to enrich queries for CLIP-based image-text retrieval.
Problem
Vision-language retrieval with CLIP-style dual encoders often confuses visually similar entities; adding textual descriptions is noisy and produces low-quality queries when naively concatenated.
Approach
Build an offline entity visual description knowledge base with LLMs, then use a fine-tuned T5 rewriter (warm-up SFT on CLIP-scored LLM rewrites, then rank-based preference optimization with CLIP scores as rewards) to select and integrate EVDs into queries that are scored alongside the original query.
Key results
- With Laion400M pre-training on Flickr30K, EvdCLIP (ViT-B/32) reaches I2T R@1 90.7 vs CLIP 89.1 and T2I R@1 75.6 vs 74.1.
- On MSCOCO 5K with Laion400M pre-training, EvdCLIP reaches I2T R@1 66.8 vs CLIP 65.3 (ViT-B/32).
- On Huawei Wallpaper T2I, EvdCLIP improves R@5 from 22.30 to 25.22 over CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LLM-generated entity visual descriptions and EaRW | CLIP ViT-B/32, Laion400M fine-tuning · Flickr30K (1K) test, I2T | R@1: 89.1 -> 90.7 | Helped |
| Add LLM-generated entity visual descriptions and EaRW | CLIP ViT-B/32, Laion400M fine-tuning · Flickr30K (1K) test, T2I | R@1: 74.1 -> 75.6 | Helped |
| Add LLM-generated entity visual descriptions and EaRW | CLIP ViT-B/32, Laion400M fine-tuning · MSCOCO (5K) test, I2T | R@1: 65.3 -> 66.8 | Helped |
| Add LLM-generated entity visual descriptions and EaRW | CLIP ViT-B/32, Laion400M fine-tuning · MSCOCO (5K) test, T2I | R@1: 48.1 -> 49.5 | Helped |
| Replace LLM-generated visual descriptions with WordNet descriptions (ablation) | EvdCLIP · Flickr30K, Laion400M, I2T | R@1: 90.7 -> 89.2 | Hurt |
| Use GPT-3 instead of ChatGPT for visual descriptions (ablation) | EvdCLIP · Flickr30K, Laion400M, I2T | R@1: 90.7 -> 90.6 | Hurt |
| Use Llama-13B instead of ChatGPT for visual descriptions (ablation) | EvdCLIP · Flickr30K, Laion400M, I2T | R@1: 90.7 -> 90.4 | Hurt |
| Use Vicuna-13B instead of ChatGPT for visual descriptions (ablation) | EvdCLIP · Flickr30K, Laion400M, I2T | R@1: 90.7 -> 90.2 | Hurt |
| Replace LLM visual entity extraction with Part of Speech Tagging (ablation) | EvdCLIP · Flickr30K, Laion400M, I2T | R@1: 90.6 -> 90.2 | Hurt |
| Replace LLM visual entity extraction with Named Entity Recognition (ablation) | EvdCLIP · Flickr30K, Laion400M, I2T | R@1: 90.6 -> 89.8 | Hurt |
| Vary the number of descriptions per entity; the paper reports best performance at H=5 (ablation) | EvdCLIP · Flickr30K | retrieval performance | Mixed |
Surprising findings
- WordNet-based descriptions give only slight gains (WordNetCLIP R@1 89.2 vs CLIP 89.1 on Flickr30K), suggesting conceptual definitions are less useful than visual descriptions.
- Vicuna-13B and Llama-2-13B generate more distracting and hallucinated descriptions than GPT-3/ChatGPT, yielding smaller gains.
Limitations
- Depends on quality of the EVD dictionary, which is generated by LLMs and may contain hallucinations.
- Over-describing less important entities in multi-entity queries can confuse the model.
- Expanded queries can exceed CLIP's 77-token limit.
Experimental setup
Datasets: Flickr30K MSCOCO Laion400M Laion-2B Merged-2B YFCC15M CC12M MSR-VTT SBU30k Huawei business datasets
Benchmarks: Flickr30K (1K) MSCOCO (5K) Huawei business datasets (Theme, Wallpaper, Lock-Screen, Icons)
Metrics: R@1 R@5 R@10 R@50 R@100
Baselines: CLIP DetCLIP DesCLIP CLIP-GPT LaBo CoCa EVA-02-CLIP WordNetCLIP
Builds on: CLIP T5 LaBo DesCLIP Bradley-Terry model
Closest papers by meaning
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.89)
- EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension (CVPR 2024 · similarity 0.88)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.87)
- SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge (NeurIPS 2024 · similarity 0.87)
- The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.87)
- CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.