Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SaCon, a symmetrical sentence- and label-anchored contrastive pre-training of two BERT encoders on Wikipedia/Wikidata relation data, which boosts several FSRE baselines on FewRel 1.0/2.0 and zero-shot RE.
Problem
Few-shot relation extraction relies on pre-trained representations; existing contrastive pre-training uses single-view or biased multi-view instance-label alignment, which limits representation quality and generalization to domain shift and unseen relations.
Approach
Two isomorphic BERT encoders (sentence and label) are pre-trained with a symmetrical contrastive loss, where sentences anchor labels and labels anchor sentences, plus MLM, so that instance and label views mutually calibrate a uniform, aligned representation space.
Key results
- SimpleFSRE + SaCon reaches 98.17% on FewRel 1.0 5-way-1-shot validation, with consistent gains over all FSRE baselines.
- On FewRel 2.0 domain adaptation, Proto-BERT + SaCon improves 5-way-1-shot accuracy from 40.12 to 78.30.
- In zero-shot RE, SaCon + SimpleFSRE reaches 97.23% (5-way-0-shot) and 95.20% (10-way-0-shot), outperforming MapRE by 6.58 and 13.74 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SaCon pre-training to SimpleFSRE | SimpleFSRE · FewRel 1.0, 5-way-1-shot | accuracy: 91.29 / 94.42 -> 98.17 / 97.88 | Helped |
| Add SaCon pre-training to Proto-BERT | Proto-BERT · FewRel 2.0 domain adaptation, 5-way-1-shot | accuracy: 40.12 -> 78.30 | Helped |
| Add SaCon pre-training to SimpleFSRE | SimpleFSRE · FewRel 2.0 domain adaptation, 5-way-5-shot | accuracy: 89.99 -> 90.32 | Mixed |
| Add SaCon pre-training to Proto-BERT | Proto-BERT · FewRel 1.0 validation, 5-way-1-shot | accuracy: MapRE 76.48; LPD 92.01; SaCon 92.38 | Helped |
| Add SaCon pre-training for zero-shot relation extraction | SimpleFSRE · FewRel 1.0 validation, 5-way-0-shot | accuracy: 90.65 -> 97.23 | Helped |
| Add label-anchored contrastive learning to sentence-anchored contrastive learning (ablation) | SaCon + SimpleFSRE · FewRel 1.0 validation, 5-way-1-shot | accuracy: 97.65 -> 98.17 | Helped |
| Add label-anchored contrastive learning to sentence-anchored contrastive learning (ablation) | SaCon + SimpleFSRE · FewRel 1.0 validation, 5-way-5-shot | accuracy: 97.76 -> 97.98 | Helped |
| Add label-anchored contrastive learning to sentence-anchored contrastive learning (ablation) | SaCon + SimpleFSRE · FewRel 1.0 validation, 10-way-1-shot | accuracy: 95.59 -> 96.21 | Helped |
| Add label-anchored contrastive learning to sentence-anchored contrastive learning (ablation) | SaCon + SimpleFSRE · FewRel 1.0 validation, 10-way-5-shot | accuracy: 95.91 -> 96.46 | Helped |
| Add sentence-anchored contrastive learning to label-anchored contrastive learning (ablation) | SaCon + SimpleFSRE · FewRel 1.0 validation, 5-way-1-shot | accuracy: 95.19 -> 98.17 | Helped |
| Add sentence-anchored contrastive learning to label-anchored contrastive learning (ablation) | SaCon + SimpleFSRE · FewRel 1.0 validation, 5-way-5-shot | accuracy: 97.84 -> 97.98 | Helped |
| Add sentence-anchored contrastive learning to label-anchored contrastive learning (ablation) | SaCon + SimpleFSRE · FewRel 1.0 validation, 10-way-1-shot | accuracy: 93.33 -> 96.21 | Helped |
Surprising findings
- Label-anchored and sentence-anchored losses alone each underperform the combined SaCon, but the gap varies by setting, e.g., sentence-anchored is better on 5-way-1-shot than label-anchored.
Limitations
- Pre-training assumes one-to-one sentence-label pairs; many-to-many relations between labels and instances are not handled.
- Gains on FewRel 1.0 are reported on validation sets for some settings and depend on baseline implementations.
Experimental setup
Datasets: Wikipedia/Wikidata pre-training corpus (867,278 sentences, 744 relations) FewRel 1.0 FewRel 2.0
Benchmarks: FewRel 1.0 FewRel 2.0 domain adaptation
Metrics: accuracy alignment uniformity
Baselines: Proto-BERT BERT-PAIR REGRAB HCRP SimpleFSRE MapRE LPD MTB CP
Builds on: MTB CP MapRE BERT Prototypical network
Closest papers by meaning
- Enhancing Low-Resource Relation Representations through Multi-View Decoupling (AAAI 2024 · similarity 0.85)
- A Positive-Unlabeled Metric Learning Framework for Document-Level Relation Extraction with Incomplete Labeling (AAAI 2024 · similarity 0.85)
- COMM: Concentrated Margin Maximization for Robust Document-Level Relation Extraction (AAAI 2025 · similarity 0.85)
- Prototype-Guided Multimodal Relation Extraction based on Entity Attributes (AAAI 2025 · similarity 0.85)
- Few-Shot, No Problem: Descriptive Continual Relation Extraction (AAAI 2025 · similarity 0.84)
- DocKS-RAG: Optimizing Document-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.