Research map / Large language models / Agents, code and math
Enhancing Relation Extraction via Supervised Rationale Verification and Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRVF, an automated feedback framework for LLM-based relation extraction where a BERT-based rationale supervisor, trained contrastively on causally collected biased/unbiased rationales, verifies predictions and retrieves re-selected demonstrations as feedback for iterative correction.
Problem
LLM-based relation extraction suffers from relation bias, and existing automated feedback methods (designed for reasoning tasks) do not fit RE because their feedback objectives and correction manner are task-specific and they omit demonstrations from feedback.
Approach
Collect unbiased rationales via label-guided causal intervention and biased rationales via diversified intervention (demonstrations with different labels), train a BERT-based rationale supervisor with contrastive loss to verify rationales, and when a rationale is biased, retrieve similar labeled demonstrations as feedback for LLM re-generation in an iterative loop.
Key results
- With Llama-2-7b-chat and task-specific retriever, SRVF averages 51.65 micro-F1 across settings vs 48.41 for in-context learning and 48.91 for the best baseline (Self-Consistency).
- On random-demonstration backbone, SRVF improves average micro-F1 by 10.65 points over in-context learning (42.25 vs 31.60).
- With Meta-Llama-3-70B-Instruct, SRVF reaches 56.94 average micro-F1 versus 54.47 for in-context learning, and is 11.27 points above KnowPrompt on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SRVF to Random ICL | Llama-2-7b-chat · SemEval, TACRED, Re-TACRED | micro-F1: 31.60 -> 42.25 | Helped |
| Add SRVF to SimCSE ICL | Llama-2-7b-chat · SemEval, TACRED, Re-TACRED | micro-F1: 43.36 -> 49.85 | Helped |
| Add SRVF to Task-specific ICL | Llama-2-7b-chat · SemEval, TACRED, Re-TACRED | micro-F1: 48.41 -> 51.65 | Helped |
| Add SRVF rather than using GRACE | Random ICL with Llama-2-7b-chat · SemEval, TACRED, Re-TACRED | micro-F1: 33.91 -> 42.25 | Helped |
| Add label-guided intervention for unbiased rationale collection (ablation) | SRVF · SemEval, TACRED, Re-TACRED | micro-F1: 36.77 -> 44.34 | Helped |
| Use diversified intervention instead of random sampling for biased rationale collection (ablation) | SRVF · SemEval, TACRED, Re-TACRED | micro-F1: 42.39 -> 44.34 | Helped |
| Use rational contrastive training instead of cross-entropy (ablation) | SRVF · SemEval, TACRED, Re-TACRED | micro-F1: 43.88 -> 44.34 | Helped |
| Retrieve feedback demonstrations rather than reuse initial demonstrations (ablation) | SRVF · SemEval, TACRED, Re-TACRED | micro-F1: 41.89 -> 44.34 | Helped |
| Regenerate the prediction using feedback demonstrations (ablation) | SRVF · SemEval, TACRED, Re-TACRED | micro-F1: 40.79 -> 44.34 | Helped |
| Set contrastive-training temperature to 0.01 or 0.05 rather than a moderate value (ablation) | SRVF with Task-specific retriever · SemEval, TACRED, Re-TACRED | micro-F1: 35.89; 30.15 | Hurt |
| Set contrastive-training temperature to 0.50 instead of 0.20 (ablation) | SRVF with Task-specific retriever · SemEval, TACRED, Re-TACRED | micro-F1: 40.68 -> 41.49 | Helped |
| Increase feedback demonstrations from 1 to 3 (ablation) | SRVF · SemEval | micro-F1 | Helped |
Surprising findings
- Self-Refine, a multi-agent feedback method, performs worst among the compared methods, below plain in-context learning on most settings.
- Adding more than about 7 feedback demonstrations degrades performance, suggesting noisy feedback.
Limitations
- Relies on LLM-generated rationales and a BERT supervisor; unbiased rationale quality is only moderate on TACRED (about 3.5/5 by GPT-4)
- Performance on document-level RE remains low (around 8-21 micro-F1), and gains are limited there
- Feedback demonstrations may contain noise, with performance dropping when too many are used
Experimental setup
Datasets: SemEval TACRED Re-TACRED DocRED Re-DocRED ACE05
Benchmarks: SemEval TACRED Re-TACRED DocRED Re-DocRED ACE05
Metrics: micro-F1
Baselines: In-context Learning Self-Refine Self-Consistency GRACE R-BERT KnowPrompt
Builds on: Self-Refine Self-Consistency GRACE SimCSE KnowPrompt
Closest papers by meaning
- Reliable Data Generation and Selection for Low-Resource Relation Extraction (AAAI 2024 · similarity 0.87)
- Large Scaled Relation Extraction With Reinforcement Learning (AAAI 2018 · similarity 0.87)
- Enhancing Low-Resource Relation Representations through Multi-View Decoupling (AAAI 2024 · similarity 0.86)
- Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models (AAAI 2025 · similarity 0.86)
- Continual Relation Extraction via Sequential Multi-Task Learning (AAAI 2024 · similarity 0.86)
- DocKS-RAG: Optimizing Document-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.