Research map / Vision-language and multimodal / Human and embodied modeling
Prototype-Guided Multimodal Relation Extraction based on Entity Attributes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PG-MRE, which adds LLM-generated entity explanations and two prototype modules (attribute and relation) to improve multimodal relation extraction on MNRE, reporting new state-of-the-art accuracy.
Problem
Multimodal relation extraction struggles when text is ambiguous or contains unseen entities and when visual appearances vary widely within or overlap across relation classes, yielding less-informative contexts.
Approach
Use LLM-generated entity explanations as semantically invariant attribute cues, condense them into cluster-level attribute prototypes (VQ-style discrete codebook with momentum updates), and use attribute-guided cross-attention to fuse visual features into relation prototypes for compact, distinctive multimodal representations.
Key results
- On the MNRE test set, PG-MRE reaches 96.34% accuracy and 91.82 F1, versus 90.48 accuracy / 89.05 F1 for the prior best TMR.
- Adding entity explanations raises F1 from 88.29 (base) to 90.39; the full model with APM and RPM reaches 91.82 F1.
- On the unseen-entity subset MNRE US, PG-MRE gets 95.81% accuracy vs 93.71% for TMR; on MNRE UT (untokenizable entities) it gets 96.00% vs 94.49%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add detailed entity explanations (ablation) | PG-MRE · MNRE testing set | F1: 88.29 -> 90.39 | Helped |
| Add the Attribute Prototype Module (APM) (ablation) | PG-MRE with entity explanations · MNRE testing set | F1: 90.39 -> 90.78 | Helped |
| Add the Relation Prototype Module (RPM) (ablation) | PG-MRE with entity explanations · MNRE testing set | F1: 90.39 -> 91.08 | Helped |
| Add both APM and RPM (ablation) | PG-MRE with entity explanations · MNRE testing set | F1: 90.39 -> 91.82 | Helped |
| Use PG-MRE instead of TMR | MNRE testing set · MNRE testing set | F1: 89.05 -> 91.82 | Helped |
| Use PG-MRE instead of TMR | MNRE US · MNRE US | accuracy: 93.71 -> 95.81 | Helped |
| Use PG-MRE instead of TMR | MNRE UT · MNRE UT | accuracy: 94.49 -> 96.00 | Helped |
| Vary the number of attribute prototypes; 23 performed best (ablation) | PG-MRE APM · MNRE testing set | F1 | Helped |
| Vary the number of relation prototypes; 23 performed best (ablation) | PG-MRE RPM · MNRE testing set | F1 | Helped |
Surprising findings
- Explanations from a small model (Qwen1.5-1.8B) performed within the normal range of larger LLMs, with GPT-4o mini best.
Limitations
- Relies on LLM-generated explanations, which may be noisy (3.34% of explanations contain noisy tokens) and are not retrieval-grounded.
- Evaluated only on the MNRE dataset.
- Hyperparameters such as prototype count were chosen by ablation on the test set.
Experimental setup
Datasets: MNRE
Benchmarks: MNRE
Metrics: Accuracy Precision Recall F1
Baselines: BERT PCNN MTB MoRe MEGA IFAformer HVPNeT TSVFN MMIB HVFormer MRE-ISE MRE
Builds on: HVPNeT TMR MEGA VQ-VAE
Closest papers by meaning
- A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.90)
- A Hierarchical Network for Multimodal Document-Level Relation Extraction (AAAI 2024 · similarity 0.89)
- Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach (NeurIPS 2024 · similarity 0.88)
- Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition (AAAI 2025 · similarity 0.87)
- DocKS-RAG: Optimizing Document-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning (ICML 2025 · similarity 0.86)
- Tackling Uncertain Correspondences for Multi-Modal Entity Alignment (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.