Research map / Graph learning / Knowledge and temporal graphs
Span Graph Transformer for Document-Level Named Entity Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Span Graph Transformer (SGT), which combines retrieved context sentences encoded jointly by BERT with a document-level span graph processed by a Graph Transformer with relative position encoding, for document-level NER.
Problem
Sentence-level BERT-based NER cannot capture long-range document-level dependencies between entity mentions, which hurts accuracy especially in low-resource distantly supervised settings.
Approach
Retrieve top-relevant context sentences by span overlap and Gaussian-weighted distance for token-level sharing, then build a sparse document-level span graph connecting nested spans within sentences and identical spans across sentences, and model it with a Graph Transformer using relative position encodings (sentence identity, start/end offset differences) as attention biases.
Key results
- SGT reaches 88.98 F1 on ACE 2004, 89.15 on ACE 2005 and 83.36 on Genia, improving over prior SOTA by 0.59, 0.95 and 1.83 points respectively.
- SGT achieves 94.04 F1 on CoNLL 2003 and 91.91 on OntoNotes 5, gains of 0.54 and 0.57 points.
- In distantly supervised BC5CDR, SGT reaches 83.91 F1, exceeding the prior best (BINDER, 81.60) by about 2.3 points, with recall up 6.57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token-level context sentences for token embeddings (ablation) | SGT · Genia | F1: 82.97 -> 83.36 | Helped |
| add token-level context sentences for token embeddings (ablation) | SGT · CoNLL03 | F1: 93.52 -> 94.04 | Helped |
| add token-level context sentences for token embeddings (ablation) | SGT · BC5CDR | F1: 83.55 -> 83.91 | Helped |
| add span-level dependencies with the Graph Transformer (ablation) | SGT · Genia | F1: 80.49 -> 83.36 | Helped |
| add span-level dependencies with the Graph Transformer (ablation) | SGT · CoNLL03 | F1: 93.64 -> 94.04 | Helped |
| add span-level dependencies with the Graph Transformer (ablation) | SGT · BC5CDR | F1: 82.47 -> 83.91 | Helped |
| add both token- and span-level dependencies (ablation) | SGT · Genia | F1: 79.58 -> 83.36 | Helped |
| add both token- and span-level dependencies (ablation) | SGT · CoNLL03 | F1: 92.89 -> 94.04 | Helped |
| add both token- and span-level dependencies (ablation) | SGT · BC5CDR | F1: 81.52 -> 83.91 | Helped |
| combine sentence distance and span overlap for context selection (ablation) | SGT · Genia | F1: 83.09 -> 83.36; 83.21 -> 83.36 | Helped |
| combine sentence distance and span overlap for context selection (ablation) | SGT · CoNLL03 | F1: 93.92 -> 94.04; 93.87 -> 94.04 | Helped |
| use a fully connected span graph instead of the proposed sparse connections (ablation) | SGT · Genia | F1: 83.36 -> 81.96 | Hurt |
Surprising findings
- Removing span-level graph modeling hurts more than removing token-level context on Genia and BC5CDR (2.87 vs 0.39 F1 drop on Genia), suggesting span interaction matters more in biomedical domains.
- A fully connected span graph performs worse than the sparse graph with nested and identical-span edges, due to quadratic edge growth and diluted attention.
Limitations
- Relies on a constituency parser to filter candidate spans, adding a dependency on an external parser.
- Context length is limited by BERT's 512 sub-token input limit, so long-range context is retrieved selectively rather than fully modeled.
Experimental setup
Datasets: ACE 2004 ACE 2005 Genia CoNLL 2003 OntoNotes 5 BC5CDR
Benchmarks: ACE 2004 ACE 2005 Genia CoNLL 2003 OntoNotes 5 BC5CDR
Metrics: precision recall F1
Baselines: Biaffine Seq2Seq (Yan et al. 2021) Locate&Label Parsing-Tree W2NER BS (Boundary Smoothing) Biaffine-CNN GPT-NER DiffusionNER BINDER DocL-NER AutoNER
Builds on: biaffine NER (Yu et al. 2020) Graphormer Transformer T5 relative position bias Boundary Smoothing
Closest papers by meaning
- Small Language Model Makes an Effective Long Text Extractor (AAAI 2025 · similarity 0.93)
- An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction (AAAI 2024 · similarity 0.87)
- SENCR: A Span Enhanced Two-Stage Network with Counterfactual Rethinking for Chinese NER (AAAI 2024 · similarity 0.87)
- UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR 2024 · similarity 0.87)
- ConsistNER: Towards Instructive NER Demonstrations for LLMs with the Consistency of Ontology and Context (AAAI 2024 · similarity 0.87)
- GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.