Research map / Vision-language and multimodal / Human and embodied modeling
Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MQSPN, a unified GMNER framework that uses multi-grained learnable queries and set prediction with Hungarian bipartite matching to jointly extract entity spans, types, and grounded regions.
Problem
Grounded multimodal NER (extracting entity spans, types, and visual regions from sentence-image pairs) suffers in unified MRC and sequence-generation methods, which struggle with ambiguous entities (intra-entity) and suffer exposure bias from autoregressive decoding (inter-entity).
Approach
Combine learnable entity-grained queries with prompt-derived type-grained queries to model intra-entity links, then reformulate GMNER as non-autoregressive set prediction trained with Hungarian bipartite matching to model inter-entity relations globally; a query-guided fusion net mediates text and visual region integration.
Key results
- On Twitter-GMNER, MQSPN reaches 58.76 GMNER F1, outperforming TIGER (56.63) by +2.13.
- On Twitter-FMNERG, MQSPN achieves 48.57 overall F1 vs. TIGER's 47.20.
- With RoBERTa-large and ViT-L/14, MQSPN reaches 61.12 GMNER F1 on Twitter-GMNER (+4.49 over TIGER).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MQSPN instead of TIGER | GMNER models · Twitter-GMNER | GMNER F1: 56.63 -> 58.76 | Helped |
| use MQSPN instead of TIGER | GMNER models · Twitter-GMNER | MNER F1: 80.28 -> 80.43 | Helped |
| use MQSPN instead of TIGER | GMNER models · Twitter-GMNER | EEG F1: 61.26 -> 62.40 | Helped |
| use MQSPN instead of TIGER | GMNER models · Twitter-FMNERG | GMNER F1 (All entity types): 47.20 -> 48.57 | Helped |
| add prompt-based type-grained queries (ablation) | MQSPN · Twitter-GMNER | GMNER F1: 56.95 -> 58.76; 1.81% | Helped |
| add prompt-based type-grained queries (ablation) | MQSPN · Twitter-FMNERG | GMNER F1: 47.23 -> 48.57; 1.34% | Helped |
| add learnable entity-grained queries instead of human-designed queries (ablation) | MQSPN · Twitter-GMNER | GMNER F1: 56.02 -> 58.76; +2.74% | Helped |
| add learnable entity-grained queries instead of human-designed queries (ablation) | MQSPN · Twitter-FMNERG | GMNER F1: 45.39 -> 48.57; +3.18% | Helped |
| add query-text cross-attention (ablation) | MQSPN QFNet · Twitter-GMNER | GMNER F1: 57.21 -> 58.76; 1.55% | Helped |
| add query-region prefix integration (ablation) | MQSPN QFNet · Twitter-FMNERG | GMNER F1: 48.33 -> 48.57; 0.24% | Helped |
| add similarity-aware aggregator (ablation) | MQSPN QFNet · Twitter-GMNER | GMNER F1: 58.13 -> 58.76; 0.63% | Helped |
| use bipartite matching loss instead of fixed-permutation joint cross-entropy loss (ablation) | MQSPN MSP · Twitter-GMNER | GMNER F1: 56.58 -> 58.76; 2.18% | Helped |
Surprising findings
- Fine-tuning the region proposal network on entity-type bounding-box annotations lowers performance rather than improving it.
- Human-designed type queries replacing learnable queries cause a 2.74-point F1 drop on Twitter-GMNER, larger than removing the type-grained prompt component.
Limitations
- MNER-QG was reproduced by the authors since its code was not released, so its numbers may not match the original.
- Comparisons exclude large language model and knowledge-enhanced methods.
Experimental setup
Datasets: Twitter-GMNER Twitter-FMNERG
Benchmarks: Twitter-GMNER Twitter-FMNERG
Metrics: precision recall F1
Baselines: GVATT UMT UMGF ITA BARTMNER H-Index TIGER MNER-QG
Builds on: TIGER H-Index MNER-QG Seq2Set PIQN prefix tuning
Closest papers by meaning
- A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.88)
- Prototype-Guided Multimodal Relation Extraction based on Entity Attributes (AAAI 2025 · similarity 0.87)
- Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach (NeurIPS 2024 · similarity 0.87)
- MCL-NER: Cross-Lingual Named Entity Recognition via Multi-View Contrastive Learning (AAAI 2024 · similarity 0.87)
- Hierarchical Aligned Multimodal Learning for NER on Tweet Posts (AAAI 2024 · similarity 0.86)
- UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.