Research map / Large language models / Language, safety and interpretability
ConsistNER: Towards Instructive NER Demonstrations for LLMs with the Consistency of Ontology and Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConsistNER, a three-stage ICL framework for low-resource NER that retrieves demonstrations by ontological consistency (ontology distribution filtering) and contextual consistency (entity-aware self-attention), using ChatGPT.
Problem
Low-resource named entity recognition with in-context learning in LLMs, where demonstration retrieval ignores the consistency of entity ontology and sentence context.
Approach
Pre-recognize entities zero-shot with an LLM, then retrieve per-sentence demonstrations by filtering on ontology distribution (bag of ontology) and ranking by entity-aware contextual representations from dual train/target-level self-attention over BERT embeddings, with a replacement step to mitigate pre-recognition error propagation.
Key results
- Under 1-shot, ConsistNER improves over strongest baselines by up to 6.01%, 26.37%, 19.56% and 21.44% F1 on CoNLL2003, OntoNotes5.0, NCBI and BC5CDR.
- Under 5-shot, gains of up to 3.07% (CoNLL2003), 9.69% (NCBI) and 21.18% (BC5CDR) over strongest baselines.
- Using ground-truth pre-recognition as a theoretical boundary yields 1.79%-5.64% improvement over empirical results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ConsistNER instead of CLS-based retrieval with the vanilla query | ChatGPT NER · CoNLL2003, 1-shot | Micro-F1: 70.66±0.2 -> 73.01±0.1 | Helped |
| Use ConsistNER instead of CLS-based retrieval with the vanilla query | ChatGPT NER · NCBI, 1-shot | Micro-F1: 27.23±0.1 -> 34.85±0.2 | Helped |
| Use ConsistNER instead of CLS-based retrieval with GPT-NER* query | ChatGPT NER · BC5CDR, 1-shot | Micro-F1: 56.73±0.3 -> 58.69±0.2 | Helped |
| Use ConsistNER instead of CLS-based retrieval with the vanilla query | ChatGPT NER · CoNLL2003, 5-shot | Micro-F1: 75.49±0.1 -> 78.87±0.1 | Helped |
| Add ontology-based demonstration filtering (ablation) | ConsistNER · CoNLL2003, 1-shot | Micro-F1: 71.12 -> 73.01 | Helped |
| Add ontology-based demonstration filtering (ablation) | ConsistNER · NCBI, 1-shot | Micro-F1: 32.97 -> 34.85 | Helped |
| Add ontology filtering and entity-aware self-attention (ablation) | ConsistNER · NCBI, 1-shot | Micro-F1: 27.23 -> 34.85 | Helped |
| Add contextual consistency retrieval (ablation) | ConsistNER · NCBI, 1-shot | Micro-F1: 25.54 -> 34.85 | Helped |
| Add demonstration replacement (ablation) | ConsistNER · BC5CDR, 1-shot | Micro-F1: 55.15 -> 57.98 | Helped |
| Replace some retrieved demonstrations with top-k semantically similar examples (ablation) | ConsistNER · CoNLL2003 | Micro-F1: 0 20 40 60 80 100 (%) | Mixed |
Surprising findings
- Erroneously pre-recognized training examples can still help because the labels in demonstrations are correct (the 'chaos phenomenon'), suggesting pre-recognition accuracy is less critical as long as same-type entities receive the same label.
- Ontology-based filtering outperforms entity-type-based filtering as the number of demonstrations increases, though entity-type filtering is better when demonstrations are scarce.
Limitations
- Performance is bounded by zero-shot LLM pre-recognition quality, causing error propagation.
- Requires manually defined ontology mapping schemas for some datasets (OntoNotes5.0 and NCBI).
- Evaluated only with ChatGPT (gpt-3.5-turbo), a closed API model.
Experimental setup
Datasets: CoNLL2003 OntoNotes5.0 NCBI BC5CDR
Benchmarks: CoNLL2003 OntoNotes5.0 NCBI BC5CDR
Metrics: span-level Micro-F1
Baselines: ProtoBERT NNShot StructShot CONTaiNER COPNER GPT-NER CLS retrieval Random retrieval
Builds on: GPT-NER prototypical networks kNN demonstration retrieval bag of words
Closest papers by meaning
- GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition (AAAI 2025 · similarity 0.90)
- UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR 2024 · similarity 0.89)
- MCL-NER: Cross-Lingual Named Entity Recognition via Multi-View Contrastive Learning (AAAI 2024 · similarity 0.88)
- A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis (AAAI 2026 · similarity 0.87)
- Span Graph Transformer for Document-Level Named Entity Recognition (AAAI 2024 · similarity 0.87)
- Small Language Model Makes an Effective Long Text Extractor (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.