Research map / Large language models / Language, safety and interpretability
STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAR, an inverse structure-to-text pipeline that uses GPT-3.5/GPT-4 to generate target structures and then passages with LLM self-reflection and self-refinement, producing training data for low-resource event extraction and relation extraction.
Problem
Low-resource information extraction, where supervised models need (passage, structure) training pairs that are costly to annotate, and existing LLM data generation methods either need ground-truth passages/labels or fail on complex structured tasks like event extraction.
Approach
Generate the target structure first (from sampled trigger and argument candidates with controlled distributions), then generate a passage conditioned on it using instruction-guided prompts, and iteratively fix errors via LLM self-reflection questions and template-based hindsight feedback, so no ground-truth passages or structures are needed.
Key results
- On ACE05 with k=10, STAR (GPT-4) data lifts DEGREE argument classification F1 by 12.91 points over no augmentation, and exceeds human-curated data at 5+ demonstrations for most metrics.
- Supervised OneIE/DEGREE trained with STAR data outperform the best inference-only GPT-4 method by at least 27.51 points F1 on argument classification.
- On TACRED, STAR (GPT-3.5) data improves SURE by 5.4 F1 and GenPT by 3.6 F1 over N=0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train DEGREE on STAR-generated data instead of no augmented data | DEGREE · ACE05, k=10 | Argument classification F1: 40.08 -> 52.99 | Helped |
| Train OneIE on STAR-generated data instead of no augmented data | OneIE · ACE05, k=10 | Argument classification F1: 33.56 -> 43.25 | Helped |
| Use STAR-generated data instead of human-curated data | DEGREE · ACE05, k=10 | Argument classification F1: 50.09 -> 52.99 | Helped |
| Use STAR-generated data instead of human-curated data | OneIE · ACE05, k=10 | Argument classification F1: 41.60 -> 43.25 | Helped |
| Use STAR-generated rather than weakly supervised training data | DEGREE · ACE05, k=10 | Argument classification F1: 30.01 -> 52.99 | Helped |
| Train SURE with STAR-generated data rather than no augmentation | SURE · TACRED, N=40 | Micro F1: 27.61 -> 33.02 | Helped |
| Train GenPT with STAR-generated data rather than no augmentation | GenPT · TACRED, N=40 | Micro F1: 33.38 -> 37.01 | Helped |
| Generate target structures with the LLM rather than use sampled human-annotated structures (ablation) | STAR with DEGREE · ACE05, k=10, N=10 | Argument classification F1: 41.77 -> 45.73 | Helped |
| Add rule-based error checking instead of no error identification (ablation) | STAR with DEGREE · ACE05, k=10, N=10 | Argument classification F1: 39.85 -> 41.26 | Helped |
| Add NLI-based self-reflection instead of no error identification (ablation) | STAR with DEGREE · ACE05, k=10, N=10 | Argument classification F1: 39.85 -> 43.54 | Helped |
| Use LLM self-reflection instead of no error identification (ablation) | STAR with DEGREE · ACE05, k=10, N=10 | Argument classification F1: 39.85 -> 45.73 | Helped |
| Use LLM self-reflection instead of rule-based error checking (ablation) | STAR with DEGREE · ACE05, k=10, N=10 | Argument classification F1: 41.26 -> 45.73 | Helped |
Surprising findings
- Generated target structures without factuality guarantees still help: diversity outweighs factuality, with structure generation beating human-annotated structures by almost 4 points Arg-C.
- Without any demonstrations (k=0), STAR-generated data still helps but remains much worse than human-curated data, indicating GPT-4 cannot produce human-level EE data without examples.
Limitations
- Experiments limited to sentence-level English EE on ACE05 and RE on TACRED with a small number of demonstrations (k up to 10).
- Relies on proprietary GPT-3.5/GPT-4 APIs, and generated structures lack guaranteed factuality.
- Quality dimensions and intervention templates are manually defined per task.
Experimental setup
Datasets: ACE05 TACRED MultiNLI
Benchmarks: ACE05 TACRED
Metrics: F1 micro F1 human quality satisfaction rate
Baselines: GPT-3.5 inference-only formulations (Text2Event, DEGREE, DICE, Task Instruction, GPT-4 inference-only (Instruction+Examples, Code4Struct) Weak supervision with LLM-predicted labels Human-curated ACE05 data SURE weak supervision
Builds on: DEGREE OneIE SURE GenPT Text2Event Code4Struct
Closest papers by meaning
- Is a Large Language Model a Good Annotator for Event Extraction? (AAAI 2024 · similarity 0.86)
- Reliable Data Generation and Selection for Low-Resource Relation Extraction (AAAI 2024 · similarity 0.86)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.85)
- GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction (ICLR 2024 · similarity 0.85)
- KGGen: Extracting Knowledge Graphs from Plain Text with Language Models (NeurIPS 2025 · similarity 0.84)
- ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.