Research map / Continual learning / Online and replay-based
Optimal Task Order for Continual Learning of Multiple Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives an analytical expression for final continual-learning error in a linear teacher-student latent-factor model, showing task order matters, and proposes periphery-to-core and max-path ordering rules validated on synthetic and image classification tasks.
Problem
Task order strongly affects continual learning of multiple tasks, but there is little understanding of how order influences final performance or how to choose an optimal order.
Approach
Uses a linear teacher-student model with shared latent factors and correlated task mixing matrices to derive a closed-form final error depending only on the upper-triangular part of the input similarity matrix, then uses linear perturbation to derive two ordering rules: periphery-to-core (least typical tasks first) and max-path (adjacent tasks dissimilar, i.e., longest Hamiltonian path in dissimil
Key results
- Analytical final-error expression matches gradient-descent simulations across random task similarity matrices and numbers of tasks.
- Periphery-to-core ordering outperforms core-to-periphery and random ordering in Fashion-MNIST, CIFAR-10, and CIFAR-100 continual image classification.
- Estimating task similarity from only 1% of training data retains a performance gain comparable to full data, while 0.1% loses significance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Order tasks from least typical to most typical | Continual image classification · Fashion-MNIST, CIFAR-10, and CIFAR-100 | average test accuracy | Helped |
| Choose the task order with the longest Hamiltonian path | Continual image classification · Fashion-MNIST, CIFAR-10, and CIFAR-100 | average test accuracy | Helped |
| Estimate task similarity using 1% of training data (ablation) | Periphery-to-core and max-path ordering · CIFAR-10 | average test accuracy | Helped |
| Estimate task similarity using 0.1% of training data (ablation) | Periphery-to-core and max-path ordering · CIFAR-10 | average test accuracy | No clear effect |
| Increase the number of binary classification tasks from 3 to 7 (ablation) | Periphery-to-core ordering · CIFAR-100 | average test accuracy | Helped |
| Increase the number of binary classification tasks from 3 to 7 (ablation) | Max-path ordering · CIFAR-100 | average test accuracy | No clear effect |
| Use periphery-to-core ordering instead of max-path ordering | Continual image classification · Fashion-MNIST, CIFAR-10, and CIFAR-100 | average test accuracy | Helped |
| Order tasks from periphery to core rather than core to periphery | Continual learning with positively correlated tasks · Linear teacher-student model | final error | Helped |
| Choose a maximum rather than minimum Hamiltonian path | Continual learning with task similarity graphs · Linear teacher-student model | final error | Helped |
| Use periphery-to-core ordering at small positive average task correlation | Continual learning with positively correlated tasks · Linear teacher-student model | final error | No clear effect |
| Use the A→C→E→D→B task order instead of A→B→C→D→E | Continual learning with chain-structured task similarity · Five-task chain | final error: approximately a seven-fold difference | Helped |
Surprising findings
- Best and worst task orders on a five-task chain similarity structure differed by roughly seven-fold in error.
- Optimal orders are non-trivial; greedy task-by-task error minimization can be suboptimal.
Limitations
- Theory derived for linear models with random task generation, limiting direct applicability.
- Each task is learned only once and trained to convergence.
- Theory requires P much less than N_x/N_s.
Experimental setup
Datasets: Fashion-MNIST CIFAR-10 CIFAR-100
Benchmarks: Fashion-MNIST CIFAR-10 CIFAR-100 synthetic linear teacher-student tasks
Metrics: final error average test accuracy after continual learning relative error
Baselines: random task order core-to-periphery order min-path order
Builds on: teacher-student linear model Hamiltonian path task ordering (longest path) outlier-task ordering in linear regression
Closest papers by meaning
- Linear Mode Connectivity in Multitask and Continual Learning (ICLR 2021 · similarity 0.90)
- Are Greedy Task Orderings Better Than Random in Continual Linear Regression? (NeurIPS 2025 · similarity 0.88)
- Disentangling and mitigating the impact of task similarity for continual learning (NeurIPS 2024 · similarity 0.88)
- Optimal Protocols for Continual Learning via Statistical Physics and Control Theory (ICLR 2025 · similarity 0.88)
- Continual Learning with Global Alignment (NeurIPS 2024 · similarity 0.88)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.