Research map / Large language models / Language, safety and interpretability

GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition

AAAI 2025 · Large language models › Language, safety and interpretability · approach: In-context learning analysis · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GuideNER, which uses LLMs to condense training-set input-label pairs into concise annotation guidelines (summarize, self-verify, aggregate top-k rules) placed in the prompt for in-context named entity recognition.

Problem

In-context learning for named entity recognition lags far behind fine-tuning, partly because demonstrations do not convey label definitions and LLMs do not learn input-label mappings from examples.

Approach

Replace retrieved examples with a static set of annotation guidelines distilled from the entire training set by the LLM itself (pattern summarization, self-verification of patterns, frequency-based top-k aggregation), giving explicit entity-range information in a shorter prompt.

Key results

• GuideNER with text-davinci-003 reaches 90.27 F1 on CoNLL03, 67.21 on ACE04, 68.58 on ACE05 and 64.76 on GENIA, beating GPT-NER on GENIA by 12.63 F1.
• GuideNER with Llama3-8B-Instruct improves over its example-based counterpart (e.g., 47.67 to 66.71 on CoNLL03) and reaches roughly text-davinci-002 level performance.
• GuideNER prompts are much shorter (506 tokens for Llama3-8B, 694 for text-davinci-003) than GPT-NER's 4,475 tokens.

Measured effects reported in the paper

Surprising findings

• Retrieving rules by semantic similarity (SimCSE) to the input performs worse than example-based prompting, due to the semantic gap between text and rules.
• Using a stronger model for summarization and a smaller model for inference still improves over example-based inference.

Limitations

• Relies on closed-source text-davinci-003 for the strongest results; the open-model Llama3-8B variant lags far behind.
• Semantic retrieval of individual rules does not work well, so the method uses the full static guideline set.
• Top-k rule selection and the number of rules needed are tuned per model, and performance declines when k is too large for the smaller model.

Experimental setup

Datasets: CoNLL03 ACE04 ACE05 GENIA

Benchmarks: CoNLL03 ACE04 ACE05 GENIA

Metrics: F1

Baselines: Llama3-8B-Instruct text-davinci-002 text-davinci-003 CodeIE Code4UIE GPT-NER C-ICL

Builds on: CodeIE GPT-NER C-ICL Code4UIE

Closest papers by meaning

• ConsistNER: Towards Instructive NER Demonstrations for LLMs with the Consistency of Ontology and Context (AAAI 2024 · similarity 0.90)
• UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR 2024 · similarity 0.89)
• A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis (AAAI 2026 · similarity 0.88)
• Entity Alignment with Noisy Annotations from Large Language Models (NeurIPS 2024 · similarity 0.88)
• MCL-NER: Cross-Lingual Named Entity Recognition via Multi-View Contrastive Learning (AAAI 2024 · similarity 0.88)
• In-Context Learning Learns Label Relationships but Is Not Conventional Learning (ICLR 2024 · similarity 0.87)