Research map / Large language models / Language, safety and interpretability
Is a Large Language Model a Good Annotator for Event Extraction?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates GPT-4, GPT-3.5-Turbo and PaLM on zero/one-shot event extraction, then uses GPT-4 (and GPT-3.5/PaLM on MAVEN) as annotators to generate labeled samples that augment ACE 2005 and MAVEN for fine-tuning event extraction models.
Problem
Event extraction suffers from data scarcity and long-tail imbalance (e.g., ACE 2005 event types with fewer than five samples), and LLMs alone perform poorly on event extraction.
Approach
Instead of using LLMs directly for extraction, prompt them with a labeled training sample as a reference to generate new labeled sentences for under-represented event types, aligning generated data distribution with the benchmark, then fine-tune standard event extraction models on the augmented data.
Key results
- Zero-shot LLM event detection on ACE 2005 is weak (GPT-4 F1 9.7 in strict mode; 52.4 in loose type-only mode), while one-shot improves GPT-4 loose F1 to 56.7.
- GPT-4-generated data (111 samples) improves CLEVE ED F1 from 71.0 to 72.8 on ACE 2005.
- For MAVEN, GPT-3.5/PaLM augmentation gives only marginal gains (e.g., 67.99% to 68.04%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GPT-4-labeled training data | BERT+CRF · ACE 2005 ED | F1: 66.4 -> 69.9 | Helped |
| Add GPT-4-labeled training data | DMBERT · ACE 2005 ED | F1: 67.7 -> 68.6 | Helped |
| Add GPT-4-labeled training data | CLEVE · ACE 2005 ED | F1: 71.0 -> 72.8 | Helped |
| Add GPT-4-labeled training data | EEQA · ACE 2005 ED | F1: 68.9 -> 69.5 | Helped |
| Add GPT-4-labeled training data | Text2Event · ACE 2005 ED | F1: 66.9 -> 67.8 | Helped |
| Add GPT-4-labeled training data | BERT+CRF · ACE 2005 EAE, gold triggers | F1: 64.5 -> 65.3 | Helped |
| Add GPT-4-labeled training data | BERT+CRF · ACE 2005 EAE, predicted triggers | F1: 56.5 -> 57.2 | Helped |
| Add GPT-4-labeled training data | DMBERT · ACE 2005 EAE, gold triggers | F1: 67.6 -> 68.4 | Helped |
| Add GPT-4-labeled training data | DMBERT · ACE 2005 EAE, predicted triggers | F1: 53.5 -> 55.5 | Helped |
| Add GPT-4-labeled training data | CLEVE · ACE 2005 EAE | F1: 72.8 -> 72.3; 60.4 -> 61.2 | Mixed |
| Add GPT-4-labeled training data | EEQA · ACE 2005 EAE, gold triggers | F1: 61.4 -> 65.2 | Helped |
| Add GPT-4-labeled training data | EEQA · ACE 2005 EAE, predicted triggers | F1: 49.6 -> 54.3 | Helped |
Surprising findings
- PaLM produced 3,182 event types not in the MAVEN event inventory during zero-shot-style extraction.
- GPT-3.5-Turbo often failed to generate a distinct sentence for annotation, requiring more explicit instructions.
Limitations
- Gap between LLM direct extraction and fine-tuned models remains large; exact-match evaluation undervalues LLM outputs.
- Augmentation gains on MAVEN are marginal, attributed to the much larger original training set and budget constraints limiting generated data.
- Only sentence-level extraction studied; LLM-generated samples may be incompletely labeled.
Experimental setup
Datasets: ACE 2005 MAVEN
Benchmarks: ACE 2005 MAVEN
Metrics: precision recall F1
Baselines: BERT+CRF DMBERT CLEVE EEQA Text2Event
Builds on: OmniEvent DYGIE++ CLEVE STAR
Closest papers by meaning
- STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models (AAAI 2024 · similarity 0.86)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.84)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.84)
- EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents (AAAI 2025 · similarity 0.84)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.84)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.