Research map / Large language models / Language, safety and interpretability
Reliable Data Generation and Selection for Low-Resource Relation Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-RDGS, which uses triplet-prompted LLM generation (ATG-DS) plus a ranking-based selection of representative sentences inside a self-supervised iterative RE training loop, for low-resource relation extraction.
Problem
Low-resource relation extraction needs annotated data, but distant supervision produces noisy sentences and prior generation methods ignore entity knowledge, yielding noisy generated data.
Approach
Generate sentences from triplets with entities as keywords and relation as constraint (ATG-DS), then select the sentence with the highest averaged cosine similarity to others in each triplet's pool using the RE model's encoder, and iterate selection with training.
Key results
- Self-RDGS with LLaMa2-7B-chat achieves 83.7% average test Micro-F1 across three low-resource datasets, versus 59.5% for RelationPrompt and 74.8% for STAD.
- MAXONE selection outperforms MINONE by 3.0 average points with GPT-2-large (81.0 vs 78.0) and 1.5 with LLaMa2-7B-chat (83.7 vs 82.2).
- On NYT10m, ATG-DS generated data with Self-RDGS reaches 74.0% test F1 versus 64.0% for high-resource EM-DS supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use triplet-guided generation and Self-RDGS instead of the high-resource supervised baseline | Relation extraction with BERTbase · NYT10m test | Micro-F1: 64.0±3.3 -> 74.0±2.0 | Helped |
| Use triplet-guided generation and Self-RDGS instead of the high-resource supervised baseline | Relation extraction with BERTbase · SemEval test | Micro-F1: 95.4±0.7 -> 88.4±0.4 | Hurt |
| Use triplet-guided generation and Self-RDGS instead of the high-resource supervised baseline | Relation extraction with BERTbase · Re-TACRED test | Micro-F1: 94.3±0.3 -> 88.7±0.6 | Hurt |
| Select the sentence with the highest global distribution score instead of using all generated sentences (ablation) | Self-RDGS with LLaMa2-7B-chat · SemEval test | Micro-F1: 86.1±0.9 -> 88.4±0.4 | Helped |
| Select the sentence with the highest global distribution score instead of using all generated sentences (ablation) | Self-RDGS with LLaMa2-7B-chat · Re-TACRED test | Micro-F1: 87.0±0.8 -> 88.7±0.6 | Helped |
| Select the sentence with the highest global distribution score instead of using all generated sentences (ablation) | Self-RDGS with LLaMa2-7B-chat · NYT10m test | Micro-F1: 70.3±2.2 -> 74.0±2.0 | Helped |
| Select the sentence with the highest rather than the lowest global distribution score (ablation) | Self-RDGS with LLaMa2-7B-chat · SemEval test | Micro-F1: 87.3±0.6 -> 88.4±0.4 | Helped |
| Select the sentence with the highest rather than the lowest global distribution score (ablation) | Self-RDGS with LLaMa2-7B-chat · Re-TACRED test | Micro-F1: 87.5±0.9 -> 88.7±0.6 | Helped |
| Select the sentence with the highest rather than the lowest global distribution score (ablation) | Self-RDGS with LLaMa2-7B-chat · NYT10m test | Micro-F1: 71.7±2.7 -> 74.0±2.0 | Helped |
| Select the highest-scoring sentence rather than the first generated sentence (ablation) | Self-RDGS with LLaMa2-7B-chat · SemEval test | Micro-F1: 87.4±0.6 -> 88.4±0.4 | Helped |
| Select the highest-scoring sentence rather than the first generated sentence (ablation) | Self-RDGS with LLaMa2-7B-chat · Re-TACRED test | Micro-F1: 87.7±0.7 -> 88.7±0.6 | Helped |
| Select the highest-scoring sentence rather than the first generated sentence (ablation) | Self-RDGS with LLaMa2-7B-chat · NYT10m test | Micro-F1: 72.1±2.1 -> 74.0±2.0 | Helped |
Surprising findings
- Fine-tuned GPT-2-large generator with Self-RDGS (81.0%) is competitive with LLaMa2-7B-chat (83.7%) and ChatGLM2-6B (80.0%) based generation.
- Vanilla in-context LLM relation prediction (LLMPREDICT) performs far below low-resource supervised training (29.6% with LLaMa2 vs 70.3% average).
Limitations
- Relies on a seed set of 20 human-annotated sentences per relation and a triplet source, so still needs some human annotation.
- Selection is based on global distribution (centrality) which assumes most generated sentences express the target relation; a small validation set is used for model selection in the baselines.
- Evaluated only on three datasets with BERT-base RE model.
Experimental setup
Datasets: SemEval-2010 Task 8 Re-TACRED NYT10m
Benchmarks: SemEval-2010 Task 8 Re-TACRED NYT10m
Metrics: Micro-F1
Baselines: SUPERVISED (low and high resource) LLMPREDICT STAD RelationPrompt MERGEALL FIRSTONE RANDONE MINONE MAXONE
Builds on: RelationPrompt distant supervision self-training (STAD) BERT entity-marker RE
Closest papers by meaning
- Improving Distantly Supervised Relation Extraction by Natural Language Inference (AAAI 2023 · similarity 0.89)
- Enhancing Relation Extraction via Supervised Rationale Verification and Feedback (AAAI 2025 · similarity 0.87)
- Distantly Supervised Entity Relation Extraction with Adapted Manual Annotations (AAAI 2019 · similarity 0.87)
- STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models (AAAI 2024 · similarity 0.86)
- Well-Written Knowledge Graphs: Most Effective RDF Syntaxes for Triple Linearization in End-to-End Extraction of Relations from Texts (Student Abstract) (AAAI 2024 · similarity 0.86)
- Enhancing Low-Resource Relation Representations through Multi-View Decoupling (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.