Research map / Vision-language and multimodal / Human and embodied modeling
A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DWE, a dual-way enhanced text-matching framework for multimodal entity linking that refines mention queries with object and facial/ANP visual attributes and enriches entity representations with Wikipedia descriptions.
Problem
Multimodal entity linking suffers from noisy image information (modality impurity) and ambiguous property-based entity representations that make entities hard to distinguish.
Approach
Treats MEL as neural text matching: mentions (enhanced by cross-modal enhancers using refined object features, facial attributes, and ANPs) are matched to entities represented by Wikipedia descriptions, with a triplet loss plus a mean-shifted contrastive cross-modal alignment loss.
Key results
- With Wikipedia entity representations, DWE reaches Top-1 of 72.5% on Richpedia, 72.8% on WikiMEL and 51.2% on Wikidiverse, reported as SOTA.
- Using refined visual features instead of primitive whole-image features raises Top-1 from 66.7% to 72.5% on Richpedia and 64.0% to 72.8% on WikiMEL.
- With 16 candidates, DWE gets 81.0% Top-1 on WikiMEL and 87.1% on Wikidiverse versus GEMEL's 75.5% and 82.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use enhanced Wikipedia entity representations instead of previous entity representations | DWE · Richpedia | Top-1 accuracy: 67.6 -> 72.5 | Helped |
| Use enhanced Wikipedia entity representations instead of previous entity representations | DWE · WikiMEL | Top-1 accuracy: 44.7 -> 72.8 | Helped |
| Use enhanced Wikipedia entity representations instead of previous entity representations | DWE · Wikidiverse | Top-1 accuracy: 47.5 -> 51.2 | Helped |
| Use enhanced entity representations | CLIP · Richpedia | Top-1 accuracy: 60.4 -> 63.9 | Helped |
| Add facial characteristics f (ablation) | DWE · Richpedia | Top-1 accuracy: 70.9 -> 72.5 | Helped |
| Add attribute-enhanced mention m_s (ANPs) (ablation) | DWE · Richpedia | Top-1 accuracy: 69.8 -> 70.9 | Helped |
| Add cross-modal alignment L_c (ablation) | DWE · Richpedia | Top-1 accuracy: 67.3 -> 69.8 | Helped |
| Add visual-enhanced feature m_v (ablation) | DWE · Richpedia | Top-1 accuracy: 63.5 -> 67.3 | Helped |
| Replace primitive visual features with refined visual features (ablation) | DWE · Richpedia | Top-1 accuracy: 66.7 -> 72.5 | Helped |
| Replace primitive visual features with refined visual features (ablation) | DWE · WikiMEL | Top-1 accuracy: 64.0 -> 72.8 | Helped |
Surprising findings
- On WikiMEL under the previous entity representation, DWE (44.7% Top-1) underperforms the CLIP baseline (58.0% with CLIP-text style dagger setting) and MMEL (71.5% Top-1), which the paper does not explain in depth.
Limitations
- Entity representations are enhanced only for the three datasets' entities using Wikipedia pages, so the gain depends on the availability of descriptions.
- Performance drops on the previous-representation WikiMEL setting relative to some baselines, indicating sensitivity to entity representation.
- Relies on external visual tools (object detector, DeepFace, Sentibank) that may introduce errors.
Experimental setup
Datasets: Richpedia WikiMEL Wikidiverse
Benchmarks: Richpedia WikiMEL Wikidiverse
Metrics: Top-1 accuracy Top-5 accuracy Top-10 accuracy Top-20 accuracy
Baselines: BLINK BERT ARNN DZMNED JMEL MEL-HI HieCoAtt GHMFC MMEL CLIP CLIP-text GEMEL
Builds on: CLIP BLINK Sentibank DeepFace triplet loss
Closest papers by meaning
- Prototype-Guided Multimodal Relation Extraction based on Entity Attributes (AAAI 2025 · similarity 0.90)
- Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach (NeurIPS 2024 · similarity 0.89)
- Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition (AAAI 2025 · similarity 0.88)
- LAFA: Multimodal Knowledge Graph Completion with Link Aware Fusion and Aggregation (AAAI 2024 · similarity 0.88)
- Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking (AAAI 2025 · similarity 0.87)
- Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.