Research map / Vision-language and multimodal / Cross-modal learning
MESED: A Multi-Modal Entity Set Expansion Dataset with Fine-Grained Semantic Classes and Hard Negative Entities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the multi-modal entity set expansion (MESE) task, the MESED dataset of 14,489 Wikipedia entities with 434,675 image-sentence pairs and fine-grained hard-negative classes, and the MultiExpan baseline pre-trained with four multi-modal objectives.
Problem
Conventional entity set expansion relies on text only and struggles with hard negative entities, synonyms, polysemous entities and long-tailed entities, and no multi-modal fine-grained ESE benchmark existed.
Approach
Expand entity sets using multi-modal (text plus image) entity representations, supported by a new fine-grained, hard-negative-rich dataset (MESED) and a MultiExpan encoder pre-trained with masked entity prediction, contrastive, clustering, and momentum distillation losses.
Key results
- MultiExpan (Full) reaches average MAP/P score 81.16 with |Seed|=3, versus 56.38 for text-only ProbExpan and 71.43 for ALBEF.
- Contrastive learning with hard negatives is the largest single gain in pre-training ablations (75.13 to 80.61 average).
- Removing images from seed entities at inference causes severe degradation (MultiExpan average drops to 50.17 when seed visual info is removed together with candidate visual info), while removing candidate visual info alone causes only slight loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MultiExpan with all four pre-training tasks instead of text-based ProbExpan | MESED |Seed|=3 · MESED |Seed|=3 | Avg: 56.38 -> 81.16 | Helped |
| Use MultiExpan with all four pre-training tasks instead of ALBEF | MESED |Seed|=3 · MESED |Seed|=3 | Avg: 71.43 -> 81.16 | Helped |
| Add contrastive learning to masked entity prediction (ablation) | MultiExpan · MESED |Seed|=3 | Avg: 75.13 -> 80.61 | Helped |
| Add clustering learning to masked entity prediction (ablation) | MultiExpan · MESED |Seed|=3 | Avg: 75.13 -> 77.70 | Helped |
| Add momentum distillation to masked entity prediction (ablation) | MultiExpan · MESED |Seed|=3 | Avg: 75.13 -> 75.78 | Helped |
| Add all three auxiliary pre-training tasks (ablation) | MultiExpan · MESED |Seed|=3 | Avg: 75.13 -> 81.16 | Helped |
| Keep both text and image inputs during pre-training (ablation) | MultiExpan · MESED |Seed|=3 | Avg: 52.70 / 61.53 -> 75.13 | Helped |
| Keep text for both seed and candidate entities during inference (ablation) | MultiExpan · MESED |Seed|=3 inference | Avg: 19.42 -> 75.13 | Helped |
| Keep text for seed entities during inference (ablation) | MultiExpan · MESED |Seed|=3 inference | Avg: 19.49 -> 73.81 | Helped |
| Keep text for candidate entities during inference (ablation) | MultiExpan · MESED |Seed|=3 inference | Avg: 73.81 -> 75.13 | Helped |
| Keep images for seed entities during inference (ablation) | MultiExpan · MESED |Seed|=3 inference | Avg: 51.38 -> 73.81 | Helped |
| Keep images for candidate entities during inference (ablation) | MultiExpan · MESED |Seed|=3 inference | Avg: 73.01 -> 75.13 | Helped |
Surprising findings
- GPT-3.5 underperforms the unsupervised CGExpan baseline, and only improves when prompted to first reason about class names (GPT+Name).
- MultiExpan performance declines on some mathematics classes where images are mainly diagrams, showing the visual modality can hurt.
Limitations
- Image utilization is limited: visual noise can exclude correct entities (15.17% promoted vs 5.17% demoted), and diagram-heavy classes like mathematics degrade.
- Only one image per sentence-entity pair is used; multiple images are left to future work.
- Image collection relies on Google Image search and automatic re-ranking, which introduces noise requiring human calibration.
Experimental setup
Datasets: MESED Wikipedia Wikidata
Benchmarks: MESED
Metrics: MAP@K P@K
Baselines: SetExpan CaSE CGExpan ProbExpan GPT-3.5 ViT BEiT CLIP-IMG CLIP ALBEF
Builds on: ProbExpan ALBEF BERT CLIP
Closest papers by meaning
- Tackling Uncertain Correspondences for Multi-Modal Entity Alignment (NeurIPS 2024 · similarity 0.88)
- Pseudo-Label Calibration Semi-supervised Multi-Modal Entity Alignment (AAAI 2024 · similarity 0.87)
- Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation (AAAI 2025 · similarity 0.86)
- Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning (ICLR 2025 · similarity 0.85)
- Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition (AAAI 2025 · similarity 0.85)
- HLMEA: Unsupervised Entity Alignment Based on Hybrid Language Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.