Research map / Continual learning / Domain and prompt-based
HOP to the Next Tasks and Domains for Continual Learning in NLP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOP, a continual learning method for NLP that combines per-problem adapters on a frozen BERT, high-order central moments (mean, variance, third moment) of token embeddings instead of [CLS], and per-problem MLP heads, evaluated across task- and domain-incremental setups.
Problem
Continual learning for NLP across both task-incremental (TIL) and domain-incremental (DIL) setups, where catastrophic forgetting and weak knowledge transfer hurt performance, and the [CLS] token summary may be inadequate for shifting input distributions.
Approach
Replace the [CLS]/AVG reduction with concatenated high-order central moments (p=3) of the unreduced token embeddings, process them with a problem-specific 2-layer MLP head, and use separate adapters per problem on a frozen BERT backbone, so one framework covers both TIL and DIL.
Key results
- On Adapter-BERT with TIL on DSC small, HOP reaches 85.63 mAcc (vs 83.96 for CTR), and 87.84 mAcc in DIL.
- Averaged across benchmarks and both setups, HOP on Adapter-BERT reaches 86.50 mAcc and 84.80 MF1, above CTR (84.40 / 82.63).
- HOP adds about 3% more total parameters than FT and about 8% more training time per problem; it is reported as about 24x faster than CTR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HOP instead of CTR | Adapter-BERT · Avg benchmarks TIL | mAcc: 86.79 -> 88.52 | Helped |
| Use HOP instead of CTR | Adapter-BERT · Avg benchmarks DIL | mAcc: 82.01 -> 84.48 | Helped |
| Combine HOP with CTR | CTR · DSC small TIL | mAcc: +3.7% | Helped |
| Combine HOP with CTR | CTR · DSC small DIL | mAcc: +0.6% | Helped |
| Replace [CLS] pooling with HOP (ablation) | Adapter-BERT · DSC small TIL | mAcc: 55.19 -> 85.63 | Helped |
| Replace [CLS] pooling with HOP (ablation) | Adapter-BERT · DSC small DIL | mAcc: 82.22 -> 87.84 | Helped |
| Use moment order p=2 instead of p=3 (ablation) | HOP · DSC small TIL | mAcc: 85.63 -> 83.52 | Hurt |
| Use moment order p=2 instead of p=3 (ablation) | HOP · DSC small DIL | mAcc: 87.84 -> 86.91 | Hurt |
| Use moment order p=4 instead of p=3 (ablation) | HOP · DSC small TIL | mAcc: 85.63 -> 84.47 | Hurt |
| Use moment order p=4 instead of p=3 (ablation) | HOP · DSC small DIL | mAcc: 87.84 -> 86.65 | Hurt |
| Use [CLS] instead of AVG for the first moment (ablation) | HOP · DSC small TIL | mAcc: 85.63 -> 85.47 | No clear effect |
| Use [CLS] instead of AVG for the first moment (ablation) | HOP · DSC small DIL | mAcc: 87.84 -> 87.50 | No clear effect |
Surprising findings
- The [CLS] token performs very poorly as a reduction for continual learning in TIL (55.19 mAcc on DSC small with Adapter-BERT FT-style setting), while AVG pooling gives 81.44.
Limitations
- Gains on TIL are larger than on DIL, and KT metrics (BwT, FwT, plasticity) are not always maximized.
- Hyperparameter p is chosen on validation results; moments above order 3 were ignored.
- Only supervised classification problems with BERT-based backbones are evaluated.
Experimental setup
Datasets: HL5Domains Liu3Domains Ding9Domains SemEval14 Task 4 DSC product reviews 20News MultiNLI
Benchmarks: ASC DSC small DSC full 20News NLI (MultiNLI)
Metrics: mAcc MF1 BwT FwT Forgetting Plasticity #OP #TP training time
Baselines: SDL FT L2 A-GEM DER++ EWC OWM UCL HAT CAT KAN SRK
Builds on: Adapter-BERT AdapterFusion CTR B-CL EWC A-GEM DER++ HAT
Closest papers by meaning
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.88)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.87)
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.87)
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.87)
- CLDyB: Towards Dynamic Benchmarking for Continual Learning with Pre-trained Models (ICLR 2025 · similarity 0.87)
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.