Research map / Continual learning / Domain and prompt-based
Adaptive Prompting for Continual Relation Extraction: A Within-Task Variance Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WAVE-CRE, a rehearsal-free continual relation extraction method that gives each task a prompt pool (prefix-tuning viewed as MoE) and uses Gaussian generative models to replay latent representations for a task predictor and relation classifier.
Problem
Catastrophic forgetting in continual relation extraction without memory buffers; existing prompt-based methods suffer from inaccurate prompt/task selection, forgetting in shared parameters, and poor handling of cross-task and within-task variance.
Approach
Allocate a task-specific prompt pool per task, with each prompt holding one prefix expert selected via key-query matching, and store per-relation Gaussian distributions over prompted and query representations to replay features; a task predictor is trained per relation (not per task) on sampled query features.
Key results
- On FewRel, WAVE-CRE reaches 85.0% accuracy at T10, versus 44.6 for L2P, 61.8 for EPI and 67.2 for HiDe-Prompt among rehearsal-free methods.
- On TACRED, WAVE-CRE reaches 78.7% at T10, compared with 54.8 for L2P, 63.4 for EPI and 72.6 for HiDe-Prompt.
- Task-incremental TACRED ablation shows removing the prompt pool lowers final-task accuracy from 85.2 to 83.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WAVE-CRE instead of CDec+ACA | FewRel continual relation extraction · FewRel, T10 | average accuracy (%): 84.8 -> 85.0 | Helped |
| use WAVE-CRE instead of CRE-DAS | TACRED continual relation extraction · TACRED, T10 | average accuracy (%): 79.1 -> 78.7 | Hurt |
| use WAVE-CRE instead of HiDe-Prompt | FewRel continual relation extraction · FewRel, T10 | average accuracy (%): 67.2 -> 85.0 | Helped |
| use WAVE-CRE instead of HiDe-Prompt | TACRED continual relation extraction · TACRED, T10 | average accuracy (%): 72.6 -> 78.7 | Helped |
| add a task-specific prompt pool rather than use a single task-specific prompt (ablation) | WAVE-CRE · TACRED task-incremental learning, T10 | average accuracy (%): 83.4 -> 85.2 | Helped |
| set experts per prompt to L=8 and selected prompts to K=1 instead of L=1 and K=8 (ablation) | WAVE-CRE · TACRED task-incremental learning, T10 | average accuracy (%): 85.2 -> 84.2 | Hurt |
| set experts per prompt to L=4 and selected prompts to K=2 instead of L=1 and K=8 (ablation) | WAVE-CRE · TACRED task-incremental learning, T10 | average accuracy (%): 85.2 -> 84.1 | Hurt |
| set experts per prompt to L=2 and selected prompts to K=4 instead of L=1 and K=8 (ablation) | WAVE-CRE · TACRED task-incremental learning, T10 | average accuracy (%): 85.2 -> 84.0 | Hurt |
| train a dedicated task predictor on synthesized query representations rather than use EPI's task identification | WAVE-CRE task prediction · FewRel task prediction after T10 | task prediction precision (%) | Helped |
| use a relation-level task predictor rather than HiDe-Prompt's task-level grouping | WAVE-CRE task prediction · TACRED task prediction after T10 | task prediction precision (%): 64.9 -> 81.5 | Helped |
Surprising findings
- WAVE-CRE is reported as nearly matching rehearsal-based CRE methods, even though it stores no training data; on FewRel it surpasses them at the last task.
Limitations
- Gaussian generative models are a simple choice; the authors note alternative generative models could be explored.
- Prefix experts are simple offset vectors, which the authors acknowledge limits expressiveness.
- Performance remains below rehearsal-based methods on TACRED and task prediction accuracy is inconsistent across tasks.
Experimental setup
Datasets: FewRel TACRED
Benchmarks: FewRel TACRED
Metrics: average accuracy across stages task prediction precision
Baselines: L2P HiDe-Prompt EPI EA-EMR RP-CRE CRL CRE-DAS CDec+ACA
Builds on: L2P HiDe-Prompt Prefix-tuning Sparse Mixture of Experts Mixture of experts as prefix-tuning (Le et al.)
Closest papers by meaning
- Continual Relation Extraction via Sequential Multi-Task Learning (AAAI 2024 · similarity 0.92)
- One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning (ICML 2024 · similarity 0.89)
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.88)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.88)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.87)
- Consistent Prompting for Rehearsal-Free Continual Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.