Research map / Continual learning / Online and replay-based
Continual Relation Extraction via Sequential Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CREST, a rehearsal-light continual relation extraction method with a frozen BERT encoder, GMM-based generated relation representations for distillation, and a new gradient-based multi-objective optimizer (Adaptive Unified Gradient Descent) that prioritizes new-task learning.
Problem
Catastrophic forgetting in continual relation extraction (CRE), where memory-buffer rehearsal raises privacy concerns and existing methods combine multiple losses by naive weighted summation.
Approach
Freeze the BERT encoder, fit per-relation GMMs on relation representations to synthesize replay representations for distillation, and combine the new-task and distillation gradients with a continual-learning-specific MOO update (AUGD) that preserves gradient magnitude proportions so new knowledge is prioritized.
Key results
- CREST reaches 84.8% final accuracy on FewRel and 79.4% on TACRED after 10 tasks, the latter 0.3 points above the best baseline.
- AUGD outperforms PCGrad, CAGrad, IMTL and Nash-MTL on TACRED; IMTL scores 74.3% after 10 tasks, 2.6 points below no-MOO training.
- Rehearsal-free prompt methods (L2P, DualPrompt, CODA-Prompt) perform poorly on CRE without a buffer, reaching about 11-12% final accuracy on FewRel.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CREST instead of the strongest listed baseline | Continual relation extraction · TACRED, T10 | mean overall accuracy: 79.1 -> 79.4 | Helped |
| use CREST instead of CDec+ACA | Continual relation extraction · FewRel, T10 | mean overall accuracy: 84.8 -> 84.8 | No clear effect |
| use AUGD instead of no MOO (ablation) | CREST · FewRel, T10 | mean overall accuracy: 83.3 -> 84.8 | Helped |
| use AUGD instead of no MOO (ablation) | CREST · TACRED, T10 | mean overall accuracy: 76.9 -> 79.4 | Helped |
| use AUGD instead of PCGrad (ablation) | CREST · FewRel, T10 | mean overall accuracy: 81.2 -> 84.8 | Helped |
| use AUGD instead of CAGrad (ablation) | CREST · FewRel, T10 | mean overall accuracy: 83.8 -> 84.8 | Helped |
| use AUGD instead of IMTL (ablation) | CREST · TACRED, T10 | mean overall accuracy: 74.3 -> 79.4 | Helped |
| use AUGD instead of Nash-MTL (ablation) | CREST · TACRED, T10 | mean overall accuracy: 75.8 -> 79.4 | Helped |
| add a memory buffer to prompt-based methods (ablation) | L2P, DualPrompt, and CODA-Prompt · FewRel, T10 | mean overall accuracy: 11.6 -> 62.3; 11.5 -> 75.3; 11.8 -> 82.6 | Helped |
| add a memory buffer to prompt-based methods (ablation) | L2P, DualPrompt, and CODA-Prompt · TACRED, T10 | mean overall accuracy: 9.7 -> 66.6; 11.4 -> 70.0; 12.2 -> 73.9 | Helped |
| use CREST instead of the strongest prompt baseline with a buffer | Continual relation extraction · FewRel, T10 | mean overall accuracy: 82.6 -> 84.8 | Helped |
| use CREST instead of the strongest prompt baseline with a buffer | Continual relation extraction · TACRED, T10 | mean overall accuracy: 73.9 -> 79.4 | Helped |
Surprising findings
- Off-the-shelf gradient-based MTL methods (PCGrad, IMTL, Nash-MTL) often reduce final accuracy compared with no MOO at all in continual relation extraction.
- Prompt-based rehearsal-free continual learning methods fail badly on CRE without a memory buffer.
Limitations
- Relies on a frozen BERT encoder, so the encoder cannot adapt to new relations.
- Results on TACRED are roughly on par with SOTA rather than clearly better on FewRel (84.8 vs 84.7-84.8 for CDec+ACA), and the TACRED table shows large per-task variance.
Experimental setup
Datasets: FewRel TACRED
Benchmarks: FewRel TACRED
Metrics: mean accuracy after each task
Baselines: EA-EMR CML EMAR+BERT RP-CRE CRECL CRL CRL+ACA CRE-DAS CDec+ACA L2P DualPrompt CODA-Prompt
Builds on: PCGrad CAGrad IMTL NashMTL CRL
Closest papers by meaning
- Adaptive Prompting for Continual Relation Extraction: A Within-Task Variance Perspective (AAAI 2025 · similarity 0.92)
- Few-Shot, No Problem: Descriptive Continual Relation Extraction (AAAI 2025 · similarity 0.89)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.86)
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.86)
- From RAG to Memory: Non-Parametric Continual Learning for Large Language Models (ICML 2025 · similarity 0.86)
- HOP to the Next Tasks and Domains for Continual Learning in NLP (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.