Research map / Large language models / Instruction and fine-tuning
UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills ChatGPT's open-domain named entity recognition ability into small LLaMA-based UniversalNER models via targeted, mission-focused instruction tuning on Pile passages, evaluated on a 43-dataset NER benchmark.
Problem
Smaller instruction-tuned models trail LLMs like ChatGPT on targeted downstream applications, and existing NER systems generalize poorly to arbitrary new entity types and domains.
Approach
Targeted distillation: sample diverse unlabeled web passages (Pile), have ChatGPT annotate entity mentions with open entity types without a predefined label set, then instruction-tune LLaMA with per-entity-type conversational queries and frequency-based negative sampling.
Key results
- UniNER-7B and UniNER-13B zero-shot reach 41.7% and 43.4% average F1 vs 34.9% for ChatGPT on the benchmark, and outperform Vicuna by over 30 points.
- Frequency-based negative sampling outperforms no negative sampling by 21.9 F1 and uniform sampling by 5.7 F1 on the seven-domain out-of-domain evaluation.
- UniNER-7B with supervised fine-tuning reaches 84.78% average F1 on 20 supervised datasets, beating InstructUIE-11B by 3.62 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill ChatGPT using diverse Pile passages for targeted NER instruction tuning | LLaMA-based UniNER · UniversalNER benchmark (43 datasets) | average F1: 34.9% -> 41.7%; 34.9% -> 43.4% | Helped |
| Increase model size from 7B to 13B | UniNER · UniversalNER benchmark (43 datasets) | average F1: 41.7% -> 43.4% | Helped |
| Replace entity-type labels with generated type definitions (ablation) | UniNER-7B · UniversalNER benchmark (43 datasets) | average F1: 11.8% | Hurt |
| Combine all entity types into one query and response (ablation) | UniNER-7B · UniversalNER benchmark (43 datasets) | average F1: 3.3% | Hurt |
| Remove negative entity-type sampling (ablation) | UniNER-7B · Negative sampling ablation | average F1: 53.4 -> 31.5; 21.9% | Hurt |
| Use uniform rather than frequency-based negative sampling (ablation) | UniNER-7B · Negative sampling ablation | average F1: 53.4 -> 47.7; 5.7% | Hurt |
| Continue instruction-tuned training with supervised human annotations | UniNER-7B · Out-of-domain evaluation (CrossNER and MIT datasets) | average F1: 53.4 -> 61.8 | Helped |
| Use a dataset-specific instruction template (ablation) | UniNER in the supervised setting · Supervised NER datasets | F1 | Helped |
| Allow partial matching of overlapping entity spans (ablation) | NER model evaluation · CrossNER and MIT out-of-domain datasets | F1 | Helped |
| Use entity-type definitions instead of labels (ablation) | UniNER-7B · Entity-type paraphrasing | robustness to paraphrasing | Helped |
Surprising findings
- Training a single query per entity type outperforms the all-in-one template, and definition-based queries underperform entity-type queries on standard benchmarks.
- Frequency-based negative sampling gives a 21.9 F1 gain over no negative sampling, indicating negative sampling is critical for closed-world transfer.
Limitations
- Strict entity-level F1 penalizes boundary mismatches, so zero-shot ability may be underestimated (partial-match results only partly address this).
- Teacher annotations come from ChatGPT only, so student quality is bounded by noisy LLM labels and label-definition inconsistencies.
- Benchmark removes labels like ELSE/MISC and splits documents into sentences, which departs from original dataset settings.
Experimental setup
Datasets: The Pile (50K sampled passages, ChatGPT-annotated) UniversalNER benchmark training splits (supervised setting) InstructUIE 20-dataset training data CrossNER MIT
Benchmarks: UniversalNER benchmark (43 datasets, 9 domains) 20 supervised NER datasets from InstructUIE CrossNER MIT
Metrics: entity-level micro F1 (strict) partial-match F1
Baselines: ChatGPT Vicuna-7B Vicuna-13B InstructUIE-11B BERT-base Alpaca LLaMA
Builds on: Alpaca Vicuna LLaMA InstructUIE The Pile
Closest papers by meaning
- ConsistNER: Towards Instructive NER Demonstrations for LLMs with the Consistency of Ontology and Context (AAAI 2024 · similarity 0.89)
- GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition (AAAI 2025 · similarity 0.89)
- Small Language Model Makes an Effective Long Text Extractor (AAAI 2025 · similarity 0.88)
- MCL-NER: Cross-Lingual Named Entity Recognition via Multi-View Contrastive Learning (AAAI 2024 · similarity 0.88)
- PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition (NeurIPS 2024 · similarity 0.88)
- Three Heads Are Better than One: Improving Cross-Domain NER with Progressive Decomposed Network (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.