Research map / Vision-language and multimodal / Human and embodied modeling
Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Curates a web-scale visual entity recognition dataset (REW) by using Gemini Pro to verify and correct candidate Entity-WebLI labels and to generate rationales and QA pairs, then trains a GiT-Large generative model on it.
Problem
Web-scale visual entity recognition (matching images to Wikipedia entities) suffers from noisy, large-scale training data, since entity labels from text k-NN matching on image-caption data are often wrong and cover only one entity per image.
Approach
Instead of asking a multimodal LLM to label images directly (found suboptimal), prompt it to verify candidate entities given Wikipedia summaries and original captions, correct them when wrong, and additionally generate rationales and multi-entity QA pairs for multi-task training of a generative entity recognizer.
Key results
- On OVEN entity test split, REW-47M GiT-Large reaches 29.6 HM after seen finetune vs 16.0 for PaLI-17B (+13.6), with 42x fewer parameters.
- Zero-shot Flowers accuracy rises from 79.8 (Entity-WebLI GiT-Large) to 88.2 with REW-47M; Aircraft from 27.7 to 50.3.
- Multimodal LLM correction improves visual matching by +10.5% relative (CLIP) and +13.9% (DINOv2) over candidate entities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train GiT-Large on REW-47M instead of Entity-WebLI | GiT-Large · OVEN entity test split | HM top-1 accuracy: 19.1 -> 23.6 | Helped |
| Train GiT-Large on REW-47M instead of Entity-WebLI | GiT-Large · OVEN query test split | HM top-1 accuracy: 10.4 -> 30.0 | Helped |
| Replace candidate entity targets with raw multimodal LLM outputs (ablation) | GiT-Large entity-target training · OVEN entity validation split | HM top-1 accuracy: 6.7 -> 3.2 | Hurt |
| Add multimodal LLM correction to candidate entities (ablation) | GiT-Large entity-target training · OVEN entity validation split | HM top-1 accuracy: 6.7 -> 12.7 | Helped |
| Provide Wikipedia and original caption during entity correction (ablation) | Multimodal LLM correction · OVEN entity validation split | HM top-1 accuracy: 12.7 -> 14.1 | Helped |
| Provide Wikipedia and caption for rationale generation (ablation) | LLM-generated rationale supervision · OVEN entity validation split | HM top-1 accuracy: 14.2 -> 14.7 | Helped |
| Give the generated rationale to the LLM when generating QA pairs (ablation) | LLM-generated QA supervision · OVEN query validation split | HM top-1 accuracy: 27.6 -> 29.0 | Helped |
| Give the entity name instead of the rationale during QA generation (ablation) | LLM-generated QA supervision · OVEN query validation split | HM top-1 accuracy: 27.6 -> 27.2 | Hurt |
| Add rationale loss to entity-only training (ablation) | GiT-Large multi-task training · OVEN query validation split | HM top-1 accuracy: 5.4 -> 6.7 | Helped |
| Add QA loss to entity-only training (ablation) | GiT-Large multi-task training · OVEN query validation split | HM top-1 accuracy: 5.4 -> 25.1 | Helped |
| Add both rationale and QA losses (ablation) | GiT-Large multi-task training · OVEN entity validation split | HM top-1 accuracy: 14.1 -> 16.0 | Helped |
| Add both rationale and QA losses (ablation) | GiT-Large multi-task training · OVEN query validation split | HM top-1 accuracy: 5.4 -> 28.2 | Helped |
Surprising findings
- Directly using the multimodal LLM's raw outputs as training targets gives poor performance (3.2 HM vs 14.1 for the corrected version), so the LLM is better used as a verifier than a teacher.
- Gemini Pro zero-shot reaches 13.3 HM on the OVEN entity split, which the trained model outperforms by a wide margin, showing the model is not merely distilling the LLM.
Limitations
- Relies on external knowledge bases such as Wikipedia for prompting, limiting applicability where such knowledge is scarce.
- Prompting multimodal LLMs with billions of parameters for annotation is time-consuming and computationally expensive, limiting scalability.
- OVEN benchmark has suboptimal annotations and top-1 evaluation underestimates models; best model errs more than 70% of the time.
Experimental setup
Datasets: Entity-WebLI WebLI OVEN LAION Conceptual Captions-12M Oxford Flowers Sun397 Food101 FGVC-Aircraft Sports100
Benchmarks: OVEN entity split OVEN query split Oxford Flowers Sun397 Food101 FGVC-Aircraft Sports100
Metrics: harmonic mean of seen/unseen top-1 accuracy top-1 accuracy
Baselines: GiT-Large trained on Entity-WebLI GER-ALD PaLI-17B PaLI-3B CLIPfusion CLIP2CLIP GiT-Large trained on WebLI-100M RECO
Builds on: Entity-WebLI GiT GER-ALD OVEN benchmark GENRE
Closest papers by meaning
- A Generative Approach for Wikipedia-Scale Visual Entity Recognition (CVPR 2024 · similarity 0.90)
- A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.89)
- Prototype-Guided Multimodal Relation Extraction based on Entity Attributes (AAAI 2025 · similarity 0.88)
- Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking (AAAI 2025 · similarity 0.88)
- Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition (AAAI 2025 · similarity 0.87)
- MOFI: Learning Image Representations from Noisy Entity Annotated Images (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.