Research map / Continual learning / Continual fine-tuning of LLMs
CLDyB: Towards Dynamic Benchmarking for Continual Learning with Pre-trained Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLDyB, an MDP-based dynamic benchmarking framework that uses greedy task sampling, clustering, and MCTS to generate challenging, algorithm-dependent task sequences for evaluating class-incremental learning methods built on pre-trained ViTs.
Problem
Static CL benchmarks built on pre-trained foundation models suffer from likely data contamination during pre-training and performance saturation, so they fail to reliably distinguish CL methods or reflect real-world task dynamics.
Approach
Formulate sequential task selection as an infinite-state MDP whose reward is forgetting minus plasticity, and approximate its solution with greedy class-pair-potential task sampling, functional k-means clustering of candidate tasks, and Monte Carlo tree search with truncated lookahead (L=1) to produce commonly and individually challenging task sequences from a 2,403-class data pool.
Key results
- On CLDyB sequences, the commonly challenging sequences reduce Final Acc by 26% and Final AR by 9% relative to random sequences across evaluated CL methods.
- CLDyB sequences achieve SRCC 0.964 and KRCC 0.905 with held-out rankings, versus 0.643/0.429 for CIFAR-100 and ImageNet-R.
- Among nine CL methods, HiDe-Prompt reaches 62.5% Final Acc on CLDyB versus 84.9% on held-out, while DualPrompt and PGP drop to about 42-44%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on CLDyB sequences instead of random task sequences | CL methods · CLDyB vs random sequences | Final Acc: 26% reduction | Hurt |
| Evaluate on CLDyB sequences instead of random task sequences | CL methods · CLDyB vs random sequences | Final AR: 9% reduction | Hurt |
| Use CLDyB sequences rather than CIFAR-100 or ImageNet-R for evaluation | CL methods · Held-out tasks | SRCC: 0.643 -> 0.964 | Helped |
| Use CLDyB sequences rather than CIFAR-100 or ImageNet-R for evaluation | CL methods · Held-out tasks | KRCC: 0.429 -> 0.905 | Helped |
| Add diffusion-generated animal classes to the data pool | CLDyB benchmark · Upcoming tasks from augmented data pool | Final AA | Hurt |
| Add diffusion-generated animal classes to the data pool | CLDyB benchmark · Upcoming tasks from augmented data pool | Final AR | Hurt |
| Use difficulty-level reward sampling to generate easy rather than hard sequences | CLDyB · CLDyB easy vs hard sequences | Final average accuracy | Helped |
| Use CLDyB instead of random sequence selection | CL methods · 40-task sequences | Performance | Hurt |
| Use greedy task sampling rather than random task sampling (ablation) | CLDyB · Commonly challenging task sequences | ACC and AR | Hurt |
| Use greedy task sampling rather than uniform class sampling within each dataset (ablation) | CLDyB · Commonly challenging task sequences | ACC and AR | Hurt |
| Enable functional task clustering rather than disabling it (ablation) | CLDyB · Commonly challenging task sequences | ACC and AR | Hurt |
| Use MCTS rather than selecting tasks most similar in representation to previous tasks (ablation) | CLDyB · Commonly challenging task sequences | ACC and AR | Hurt |
Surprising findings
- Advanced methods such as PGP and DualPrompt do not always outperform simpler ones in robustness (e.g., PGP vs DualPrompt, CLSER vs ER).
- The simple ER baseline remains competitive on final ALA, robustness, and memory efficiency despite weak final accuracy.
Experimental setup
Datasets: Fruits-Vegetable Jute-Pest Animal-90 Sea-Animal Rock Butterfly-75 Clothing-20 Apparel Food101 FoodX251 Indian-Food CNFOOD241
Benchmarks: CIFAR-100 ImageNet-R CLDyB sequences held-out random sequences
Metrics: Final average accuracy average retention average learning accuracy SRCC KRCC memory efficiency robustness
Baselines: ER AFEC CLSER DualPrompt LAE HiDe-Prompt SLCA RanPAC PGP
Builds on: Dynabench Monte Carlo tree search L2P DualPrompt RanPAC
Closest papers by meaning
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.89)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.88)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.88)
- Where is the Truth? The Risk of Getting Confounded in a Continual World (ICML 2025 · similarity 0.87)
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.87)
- HOP to the Next Tasks and Domains for Continual Learning in NLP (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.