Research map / Continual learning / Continual fine-tuning of LLMs
Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Conditional Attention (ablation) | BERT-BASE MTL with random task sampling · GLUE dev | Avg GLUE: 80.61 -> 82.41 | Helped |
| add Conditional Adapter (ablation) | BERT-BASE MTL with Conditional Attention · GLUE dev | Avg GLUE: 82.41 -> 82.90 | Helped |
| add Conditional Alignment and Conditional Layer Normalization (ablation) | BERT-BASE MTL with Conditional Adapter · GLUE dev | Avg GLUE: 82.90 -> 83.12 | Helped |
| add MT-Uncertainty task sampling (ablation) | BERT-BASE MTL with CA and CLN · GLUE dev | Avg GLUE: 83.12 -> 84.03 | Helped |
| replace Counterfactual task selection with MT-Uncertainty sampling | BERT-BASE MTL · GLUE | maximum average GLUE score: 1.7% | Helped |
| use MT-Uncertainty rather than random task sampling | BERT-BASE MTL · GLUE dev | average GLUE score | Helped |
| replace block-diagonal Conditional Attention with Full Block Attention (ablation) | CA-MTL BERT-LARGE with 50% layers frozen · GLUE dev | Avg GLUE: 85.5 -> 81.8 | Hurt |
| freeze 25% rather than 0% of Transformer layers (ablation) | CA-MTL BERT-LARGE · GLUE dev | Avg GLUE: 84.9 -> 85.1 | Helped |
| freeze 50% rather than 25% of Transformer layers (ablation) | CA-MTL BERT-LARGE · GLUE dev | Avg GLUE: 85.1 -> 85.5 | Helped |
| use task-conditioned alignment instead of separate alignment matrices per task | BERT-LARGE MTL on CoLA, MRPC, QNLI, RTE, and SST-2 · five GLUE tasks | average score improvement over BERT-LARGE: 3.61% | Helped |
| scale MTL from 1 to 23 tasks without CA-MTL modules | BERT-BASE MTL · GLUE average | average GLUE score: 1.8% | Hurt |
| add CA-MTL modules when scaling task count | BERT-BASE MTL · 23-task GLUE | average GLUE score versus baseline MTL: close to 4% | Helped |
Closest papers by meaning
- Parameter-Efficient Transfer Learning for NLP (ICML 2019 · similarity 0.90)
- ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (ICLR 2022 · similarity 0.89)
- AdaDARE-gamma: Balancing Stability and Plasticity in Multi-modal LLMs through Efficient Adaptation (CVPR 2025 · similarity 0.87)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.87)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.87)
- PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.