Research map / Video / Video-language understanding
A Hierarchical Network for Multimodal Document-Level Relation Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Multimodal Document-level Relation Extraction (MDocRE) with a new video-paired dataset from VOA news and a hierarchical textual-guided transformer with a mention gate, reporting gains over unimodal and multimodal baselines.
Problem
Document-level relation extraction struggles with long-range dependencies between entities and with selecting which of many entity mentions are relevant to a relation; text alone is insufficient for these issues.
Approach
Adds a video modality to DocRE and fuses it with text through a hierarchical (local phrase/intra-frame and global semantic/inter-frame) encoder using a textual-guided transformer where text features act as prompts, plus a mention gate that weights entity mentions via intra- and inter-modality attention.
Key results
- The proposed hierarchical model reaches 40.54 F1 / 37.22 Ign F1 on the MDocRE test set, above the best baseline ViViT+BERT-E+Ts at 37.45 F1.
- Adding video to unimodal ATLOP and BERT-E improves their dev F1 by about 2 points (to 40.29 and 40.35 from 38.63 and 38.12).
- Removing the hierarchical fusion framework and replacing it with non-hierarchical One-stream fusion drops F1 by 19.00 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add video information using the proposed hierarchical fusion framework to BERT-E | BERT-E · MDocRE | F1: 38.12 -> 40.35 | Helped |
| Add video information using the proposed hierarchical fusion framework to ATLOP | ATLOP · MDocRE | F1: 38.63 -> 40.29 | Helped |
| Replace hierarchical fusion with non-hierarchical textual-guided fusion (ablation) | Proposed model · MDocRE | F1: ↓3.72 | Hurt |
| Replace hierarchical fusion with non-hierarchical two-stream fusion (ablation) | Proposed model · MDocRE | F1: ↓5.55 | Hurt |
| Replace hierarchical fusion with non-hierarchical one-stream fusion (ablation) | Proposed model · MDocRE | F1: ↓19.00 | Hurt |
| Replace textual-guided transformer with visual-guided transformer (ablation) | Proposed model · MDocRE | F1: ↓11.55 | Hurt |
| Replace textual-guided transformer with hierarchical two-stream transformer (ablation) | Proposed model · MDocRE | F1: ↓4.02 | Hurt |
| Add mention gate modules (ablation) | Proposed model · MDocRE | F1: ↓2.02 | Helped |
| Add the textual mention gate (ablation) | Proposed model · MDocRE | F1: ↓1.39 | Helped |
| Add the visual mention gate (ablation) | Proposed model · MDocRE | F1: ↓1.11 | Helped |
Surprising findings
- Several unimodal baselines (ATLOP, BERT-E) outperform several multimodal baselines, which the authors attribute to noise from long visual sequences fed into inappropriate fusion architectures.
- Using visual features as guidance (Vg) in the textual-guided transformer performs worse than the standard two-stream transformer, showing textual guidance is key.
Limitations
- Struggles with overlapping relations where the same entity pair holds multiple relations; the model extracted only one of two triples in the example.
- Nearly identical mention weights in both modalities for overlapping-relation cases, indicating difficulty distinguishing mentions there.
- Visual sequences can introduce noise when fused with inappropriate architectures, so unimodal baselines can beat some multimodal ones.
Experimental setup
Datasets: MDocRE dataset (VOA news scripts and videos) DocRED
Benchmarks: MDocRE DocRED
Metrics: F1 Ign F1
Baselines: ATLOP BERT-E LSR AGGCN ClipBERT VideoMAE+BERT+Os ViViT+BERT+Ts
Builds on: BERT ATLOP ClipBERT ViLBERT VisualBERT
Closest papers by meaning
- Prototype-Guided Multimodal Relation Extraction based on Entity Attributes (AAAI 2025 · similarity 0.89)
- DocKS-RAG: Optimizing Document-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning (ICML 2025 · similarity 0.88)
- A Positive-Unlabeled Metric Learning Framework for Document-Level Relation Extraction with Incomplete Labeling (AAAI 2024 · similarity 0.87)
- COMM: Concentrated Margin Maximization for Robust Document-Level Relation Extraction (AAAI 2025 · similarity 0.86)
- Revisiting Document-Level Relation Extraction with Context-Guided Link Prediction (AAAI 2024 · similarity 0.86)
- A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.