Research map / Large language models / Language, safety and interpretability
GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Code-LLaMA (7B/13B/34B) with QLoRA to follow Python-code-formatted annotation guidelines, enabling much stronger zero-shot information extraction on unseen schemas.
Problem
LLMs fail to follow IE annotation guidelines out of the box, so zero-shot information extraction on unseen label schemas lags behind task-specific and encoder models.
Approach
Represent schemas, guidelines (as docstrings and comments), and outputs as Python code, and fine-tune an LLM with guideline-level noise (class shuffling, dropout, paraphrasing, candidate sampling, label masking) plus loss only on output tokens so the model attends to guideline text rather than memorizing label names.
Key results
- Zero-shot average F1 of GoLLIE 7B is 55.3 vs 42.3 for the baseline without guidelines (about 13 points gain), and GoLLIE 34B reaches 57.2 average over all zero-shot datasets and 60.0 on the average-SoTA subset.
- GoLLIE 34B outperforms previous zero-shot SoTA on most datasets, e.g. AI 61.6 vs 49.0 and Politics 60.2 vs 48.2.
- Hallucinated labels and unparseable outputs occur in less than 1% of predictions on most zero-shot datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add annotation guidelines to the fine-tuning and inference setup | Code-LLaMA 7B for zero-shot IE · Zero-shot average all | F1: 42.3${\scriptscriptstyle\pm}$ -> 55.3${\scriptscriptstyle\pm}$ | Helped |
| Scale the GoLLIE backbone from 7B to 34B | GoLLIE · Zero-shot average all | F1: 55.3${\scriptscriptstyle\pm}$ -> 57.2${\scriptscriptstyle\pm}$ | Helped |
| Remove class order shuffling (ablation) | GoLLIE · Zero-shot all datasets | F1: 55.3${\scriptscriptstyle\pm}$ -> 55.9${\scriptscriptstyle\pm}$ | No clear effect |
| Remove guideline paraphrasing (ablation) | GoLLIE · Zero-shot all datasets | F1: 55.3${\scriptscriptstyle\pm}$ -> 54.8${\scriptscriptstyle\pm}$ | No clear effect |
| Remove class name masking (ablation) | GoLLIE · Zero-shot all datasets | F1: 55.3${\scriptscriptstyle\pm}$ -> 54.6${\scriptscriptstyle\pm}$ | No clear effect |
| Add class dropout (ablation) | GoLLIE · Zero-shot all datasets | F1: 54.0${\scriptscriptstyle\pm}$ -> 55.3${\scriptscriptstyle\pm}$ | Helped |
| Add representative annotation candidates (ablation) | GoLLIE · Zero-shot all datasets | F1: 49.9${\scriptscriptstyle\pm}$ -> 55.3${\scriptscriptstyle\pm}$ | Helped |
| Add the GoLLIE components, including guidelines (ablation) | Code-LLaMA 7B · Zero-shot all datasets | F1: 42.3${\scriptscriptstyle\pm}$ -> 55.3${\scriptscriptstyle\pm}$ | Helped |
Surprising findings
- Class order shuffling, guideline paraphrasing, and label masking showed no significant contribution in the ablation, while removing representative candidates caused the largest drop among components.
- Training the full model with the same setup performed very poorly versus QLoRA, which overfit quickly.
Limitations
- Performance is poor on ambiguous or coarse labels such as Miscellaneous and Plot, and the model confuses fine-grained and coarse person labels.
- Datasets with hundreds of labels exceed context limits when guidelines are included.
- Pre-training data contamination of Code-LLaMA cannot be ruled out.
Experimental setup
Datasets: ACE05 BC5CDR CoNLL 2003 DIANN NCBIDisease OntoNotes 5 RAMS TACRED WNUT 2017 BroadTwitter CASIE CrossNER
Benchmarks: CrossNER MIT Movie MIT Restaurants BroadTwitter CASIE E3C FabNER HarveyNER MultiNERD WikiEvents ACE05 BC5CDR
Metrics: F1
Baselines: Instruct-UIE PromptNER Entailment based IE GPT-3.5
Builds on: Code-LLaMA QLoRA Instruct-UIE PromptNER
Closest papers by meaning
- ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking (NeurIPS 2025 · similarity 0.85)
- STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models (AAAI 2024 · similarity 0.85)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.84)
- Agent Instructs Large Language Models to be General Zero-Shot Reasoners (ICML 2024 · similarity 0.83)
- UMIE: Unified Multimodal Information Extraction with Instruction Tuning (AAAI 2024 · similarity 0.83)
- Task Contamination: Language Models May Not Be Few-Shot Anymore (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.