Research map / Continual learning / Online and replay-based
Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes H-embedding, an online task embedding distilled from H-score transferability with AHP normalization, used to guide a task-conditioned hypernetwork (full-model or LoRA generation) for rehearsal-free task-incremental continual learning.
Problem
Continual learning under rehearsal-free settings lacks prior exploitation of inter-task relationships; existing methods are model-centric and suffer negative transfer and scalability issues as task sequences grow.
Approach
Learn a low-dimensional task embedding online whose Euclidean distances match AHP-normalized, reversed H-score transferability between tasks, and use it (via an encoder-decoder guidance loss) to guide a hypernet that generates task-specific weights or LoRA parameters.
Key results
- On CIFAR-100 (N=10) with ResNet-32, H-embed Hnet reaches 83.08% FAA vs 82.75% for WSN and 81.57% for HyperNet.
- On ImageNet-R (N=10) with ViT-B/16 LoRA, H-embed Hnet-LoRA reaches 81.38% FAA vs 77.18% for SD-LoRA.
- On DomainNet (N=5), H-embed Hnet-LoRA reaches 76.64% FAA vs 72.58% for SD-LoRA; on ImageNet-R (N=20) it reaches 79.90% vs 74.05%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add H-embedding guidance (ablation) | H-embed Hnet-LoRA · ImageNet-R (N = 5, 10, 20) | FAA: 77.37 -> 79.72; 42.85 -> 81.65; 74.82 -> 76.91 | Helped |
| Add CL regularization (ablation) | H-embed Hnet-LoRA · ImageNet-R (N = 5, 10, 20) | FAA: 39.08 -> 79.72; 39.89 -> 81.65; 34.57 -> 76.91 | Helped |
| Add AHP normalization (ablation) | H-embed Hnet-LoRA · ImageNet-R (N = 5, 10, 20) | FAA: 76.37 -> 79.72; 78.37 -> 81.65; 74.72 -> 76.91 | Helped |
| Add H-embedding guidance instead of using a vanilla hypernetwork (ablation) | H-embed Hnet · PermutedMNIST (10 tasks) | FAA: 97.495 -> 97.553 | Helped |
| Add H-embedding guidance instead of using a vanilla hypernetwork (ablation) | H-embed Hnet · CIFAR-100 (10 tasks) | FAA: 69.620 -> 72.390 | Helped |
| Add H-embedding guidance instead of using a vanilla hypernetwork (ablation) | H-embed Hnet · ImageNet-R (10 tasks) | FAA: 38.202 -> 39.212 | Helped |
| Replace the H-score transferability metric with LEEP (ablation) | H-Embedding · ImageNet-R (10 tasks) | FAA: 81.8 -> 81.6 | Hurt |
| Replace the H-score transferability metric with LogME (ablation) | H-Embedding · ImageNet-R (10 tasks) | FAA: 81.8 -> 81.4 | Hurt |
| Replace H-Embedding with Task2Vec (ablation) | H-embed Hnet · ImageNet-R (10 tasks) | FAA: 81.8 -> 79.8 | Hurt |
| Use H-embed Hnet-LoRA instead of SD-LoRA | ViT-B/16 · DomainNet (N = 5) | DAA: 78.79 -> 78.26 | Hurt |
| Use H-embed Hnet-LoRA instead of SD-LoRA | ViT-B/16 · ImageNet-R (N = 5) | DAA: 82.50 -> 79.72 | Hurt |
Surprising findings
- Replacing H-score with LEEP or LogME in the embedding led only to minor FAA drops (81.6 and 81.4 vs 81.8), suggesting the transferability metric is not critical in itself.
- Task2Vec-based embeddings reduced FAA to 79.8, indicating the explicit task-to-task relational modeling matters more than the specific metric.
Limitations
- CIL extension relies on a separately trained task-ID classifier over frozen pretrained features, which is not integrated with the online framework.
- Main experiments are restricted to task-incremental learning where task IDs are available at train and test time.
- H-embedding guidance is only computed after the first two tasks, and H(T_j,T_j) is approximated via an HGR surrogate.
Experimental setup
Datasets: PermutedMNIST CIFAR-100 ImageNet-R DomainNet
Benchmarks: CIFAR-100 ImageNet-R DomainNet PermutedMNIST
Metrics: FAA DAA forward transfer backward transfer
Baselines: Full Finetune LwF EWC L2 PredKD+FeatKD PackNet HyperNet WSN L2P DualPrompt CODA-Prompt HiDe-Prompt
Builds on: hypernetworks (Oswald et al.) H-score transferability analytic hierarchy process LoRA SD-LoRA
Closest papers by meaning
- Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.88)
- Disentangling and mitigating the impact of task similarity for continual learning (NeurIPS 2024 · similarity 0.87)
- Optimal Task Order for Continual Learning of Multiple Tasks (ICML 2025 · similarity 0.87)
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.86)
- InTAct: Interval-based Task Activation Consolidation for Continual Learning (NeurIPS 2026 · similarity 0.86)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.