Research map / Vision-language and multimodal / Multimodal LLMs
UMIE: Unified Multimodal Information Extraction with Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UMIE, a FLAN-T5-based encoder-decoder that unifies multimodal NER, relation extraction, and event extraction as instruction-tuned generation, with a gated visual attention module, and reports SoTA on six MIE datasets.
Problem
Multimodal information extraction (MNER, MRE, MEE) methods are task-specific, limiting generalization across tasks and failing to share knowledge across tasks; they also rarely extract visual mentions.
Approach
Cast all three MIE tasks as text-to-structure generation with task instructions on a FLAN-T5 encoder-decoder, and add a gated cross-attention module that fuses global and object-level visual features into the decoder input through a learned scalar gate.
Key results
- UMIE-XLarge reaches 78.2 (Twitter-15), 91.4 (Twitter-17), 91.0 (SNAP) F1 on MNER, and 86.4 / 76.2 F1 on MNRE-V1 / MNRE-V2, reported as SoTA on all six datasets.
- In zero-shot setting on Twitter-17 and MNRE-V2, UMIE-XLarge gets 69.9 and 69.6 F1, above GPT-4 (66.6 and 42.1).
- On M2E2, UMIE-XLarge reaches 62.1 MED and 24.5 MEAE F1, above prior best Unicl (57.6 and 23.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Unified generation framework with instruction tuning trained across MNER, MRE, and MEE (single model) | UMIE · Six MIE datasets (supervised) | F1 | Helped |
| Scale backbone from FLAN-T5-base to large to xl | UMIE · Supervised MNER/MRE/MEE | F1: consistent gains from Base to Large to XLarge | Helped |
| Exclude target dataset from training (zero-shot) with instruction-tuned multi-task model | UMIE-XLarge · Twitter-17 and MNRE-V2 zero-shot | F1: 69.9 vs GPT-4 66.6 on Twitter-17; 69.6 vs 42.1 on MNRE-V2 | Helped |
| Gated attention with dynamic gate vs fixed gate values (g in [0,0.1]) (ablation) | UMIE-Base · Averaged over MNER, MRE, MEE | F1: dynamic gate beats any fixed gate; vs g=0 gains of 0.1 (MNER), 2.3 (MRE), 3.3 (MEE) | Helped |
| Fixed gate value above 0.1 (visual features weighted heavily) (ablation) | UMIE-Base · MNER, MRE, MEE | F1 | Hurt |
| Visual features contribution (gating) barely affects MNER (ablation) | UMIE-Base · MNER (Twitter-15/17, SNAP) | F1: average 0.1 gain over text-only | No clear effect |
| Robustness to instruction phrasing (three MNER instructor variants) (ablation) | UMIE-XLarge · Twitter-15/17, SNAP | F1: Twitter-15 78.0/77.9/78.2 across I0/I1/I2 | No clear effect |
| Add Twitter training corpus vs News corpus (ACE2005, M2E2) in training (ablation) | UMIE-Base · MNER, MRE, MEE | F1 | Mixed |
| Increase News-domain training examples (ablation) | UMIE-Base · Twitter MNER/MRE tasks | F1 | No clear effect |
Surprising findings
- Gate values are very small (effective range 0-0.1), yet fixing the gate above 0.1 significantly hurts performance across tasks.
- MNER is hardly sensitive to visual features (about 0.1% F1 gain from dynamic gate vs. text-only), while MRE and MEE gain more (2.3% and 3.3%).
Limitations
- Visual features come from an off-the-shelf grounding toolkit (up to 36 RoIs), so quality depends on that detector.
- Main results compare against reported numbers from prior papers rather than re-run baselines under identical settings.
- Zero-shot evaluation excludes only one dataset at a time, and MEE task includes MEAE with cascaded inference.
Experimental setup
Datasets: Twitter-15 Twitter-17 SNAP MNRE-V1 MNRE-V2 ACE2005 SWiG M2E2
Benchmarks: Twitter-15 Twitter-17 SNAP MNRE-V1 MNRE-V2 M2E2
Metrics: F1
Baselines: UMT UMGF MEGA RpBERT R-GCN HVPNeT ITA MoRe MNER-QG WASE Unicl ChatGPT
Builds on: FLAN-T5 UMT instruction tuning ResNet-101
Closest papers by meaning
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.86)
- Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.86)
- UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark (CVPR 2026 · similarity 0.86)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.85)
- Emu: Generative Pretraining in Multimodality (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.