Research map / Large language models / Language, safety and interpretability
Small Language Model Makes an Effective Long Text Extractor
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeNER, a lightweight span-based NER model for long texts that combines arrow attention with LogN-scaling on [CLS] and a bidirectional sliding-window plus-shaped attention (BiSPA) over token-pair span tensors, achieving SOTA on three long NER datasets with lower memory.
Problem
Named entity recognition over long texts with long entity spans (e.g., awards, work experience) is under-explored; span-based methods scale O(L^2) in memory and generation-based LLM methods struggle with long spans and are slow.
Approach
Encode long inputs with arrow attention (local sliding window plus global [CLS] attention with LogN-scaling on [CLS]) and prune the token-pair span tensor to spans up to length w', then apply bidirectional sliding-window plus-shaped attention so interactions between spans are modeled at reduced cost.
Key results
- SeNER achieves 51.56 F1 on Scholar-XL vs 49.10 for UTC-IE and 47.59 for CNN-NER, and 74.49 F1 on SciREX (best).
- On Profiling-07 SeNER gets 67.34 F1, slightly below UTC-IE's 67.43.
- SeNER supports input lengths about 3x longer than CNN-NER and 6x longer than UTC-IE; GPT-4o 5-shot gets 21.87 F1 on Scholar-XL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LogN-Scaling to the [CLS] token (ablation) | SeNER · Scholar-XL, SciREX, and Profiling-07 | F1: 50.78 -> 51.56; 74.27 -> 74.49; 67.11 -> 67.34 | Helped |
| add Arrow Attention (ablation) | SeNER · Scholar-XL | F1: 51.34 -> 51.56 | Helped |
| add Arrow Attention (ablation) | SeNER · SciREX and Profiling-07 | F1 | Helped |
| replace Arrow Attention with sliding-window attention (ablation) | SeNER · Scholar-XL, SciREX, and Profiling-07 | F1: 51.56 -> 49.95; 74.49 -> 73.50; 67.34 -> 65.60 | Hurt |
| add LoRA (ablation) | SeNER · Scholar-XL, SciREX, and Profiling-07 | F1: 49.93 -> 51.56; 73.98 -> 74.49; 66.53 -> 67.34 | Helped |
| add BiSPA (ablation) | SeNER · Scholar-XL | F1: 50.48 -> 51.56 | Helped |
| add BiSPA (ablation) | SeNER · SciREX and Profiling-07 | F1 | Helped |
| add whole word masking (ablation) | SeNER · Scholar-XL, SciREX, and Profiling-07 | F1: 50.45 -> 51.56; 74.24 -> 74.49; 64.97 -> 67.34 | Helped |
| use SeNER instead of the strongest baseline | long-text NER · Scholar-XL | F1: 49.10 -> 51.56 | Helped |
| use SeNER instead of the strongest baseline | long-text NER · SciREX | F1: 73.42 -> 74.49 | Helped |
| use SeNER instead of the strongest baseline | long-text NER · Profiling-07 | F1: 67.43 -> 67.34 | Hurt |
Surprising findings
- Prompting GPT-4o and Claude-3.5 with 5 demonstrations performs poorly on long NER (21.87 F1 on Scholar-XL).
- Replacing arrow attention with sliding window attention lowers F1 on all datasets, showing the value of global [CLS] attention.
Limitations
- Very long entities (education/work experience) are weaker than the best baselines, attributed to information loss from the span-length approximation.
- SciREX and Profiling-07 are truncated to 5120 tokens with sliding windows due to GPU memory limits.
- On Profiling-07 SeNER does not beat UTC-IE (67.34 vs 67.43 F1).
Experimental setup
Datasets: Scholar-XL SciREX Profiling-07
Benchmarks: Scholar-XL SciREX Profiling-07
Metrics: micro precision micro recall micro F1 GPU memory (GB) maximum input length inference time
Baselines: CNN-NER UTC-IE DiffusionNER UIE InstructUIE GOLLIE ADELIE ToNER GPT-4o Claude-3.5
Builds on: UTC-IE CNN-NER DeBERTa-V3 LoRA sliding window attention
Closest papers by meaning
- Span Graph Transformer for Document-Level Named Entity Recognition (AAAI 2024 · similarity 0.93)
- UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR 2024 · similarity 0.88)
- PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition (NeurIPS 2024 · similarity 0.87)
- An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction (AAAI 2024 · similarity 0.87)
- SENCR: A Span Enhanced Two-Stage Network with Counterfactual Rethinking for Chinese NER (AAAI 2024 · similarity 0.86)
- ConsistNER: Towards Instructive NER Demonstrations for LLMs with the Consistency of Ontology and Context (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.