Research map / Continual learning / Class-incremental
Continual Learning Using a Kernel-Based Method Over Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KLDA, a frozen-foundation-model class-incremental learner that applies random Fourier feature RBF kernel approximation and incrementally updated class means with a shared covariance for LDA classification, with no replay.
Problem
Class-incremental learning suffers from catastrophic forgetting and inter-task class separation (ICS); existing CIL methods remain well below the joint-training upper bound.
Approach
Keep a self-supervised foundation model frozen and, instead of training, enhance its features with an RBF kernel approximated by random Fourier features, then store only per-class means and one shared covariance matrix to perform LDA classification incrementally.
Key results
- On CLINC, Banking, DBpedia and HWU with BART-base, KLDA-E reaches 96.62, 93.03, 94.53 and 89.78% final accuracy, versus Joint Fine-tuning at 95.33, 91.36, 94.83 and 88.60% and the best no-replay baseline VAG at 76.42, 59.34, 65.40 and 56.88%.
- Across five other language models (MiniLM, BERT-base, RoBERTa-large, T5-3b, Mistral-7b), KLDA-E stays close to Joint fine-tuning on all four text datasets.
- On image data with DINOv2-base, KLDA reaches 98.45% on CIFAR10 (Joint 98.54%) but 88.81% on CIFAR100 (Joint 90.30%) and 83.18% on TinyImageNet (Joint 86.43%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the RBF kernel through RFF instead of using original FM features with LDA | LDA class-incremental classifier · CLINC | final accuracy (%): 93.71 -> 95.90 | Helped |
| Apply the RBF kernel through RFF instead of using original FM features with LDA | LDA class-incremental classifier · Banking | final accuracy (%): 89.09 -> 92.23 | Helped |
| Apply the RBF kernel through RFF instead of using original FM features with LDA | LDA class-incremental classifier · DBpedia | final accuracy (%): 93.42 -> 94.13 | Helped |
| Apply the RBF kernel through RFF instead of using original FM features with LDA | LDA class-incremental classifier · HWU | final accuracy (%): 86.41 -> 87.27 | Helped |
| Add the 5-model probability-averaging ensemble | KLDA · CLINC | final accuracy (%): 95.90 -> 96.62 | Helped |
| Add the 5-model probability-averaging ensemble | KLDA · Banking | final accuracy (%): 92.23 -> 93.03 | Helped |
| Add the 5-model probability-averaging ensemble | KLDA · DBpedia | final accuracy (%): 94.13 -> 94.53 | Helped |
| Add the 5-model probability-averaging ensemble | KLDA · HWU | final accuracy (%): 87.27 -> 89.78 | Helped |
| Use KLDA instead of joint fine-tuning | BART-base CIL classifier · CLINC, Banking, DBpedia, and HWU | final accuracy (%) | Mixed |
| Use KLDA instead of joint fine-tuning | DINOv2 image classifier · CIFAR10, CIFAR100, T-ImageNet, and Cars | final accuracy (%) | Mixed |
Surprising findings
- Plain NCM and LDA without kernels already outperform all fine-tuning and pseudo-replay CIL baselines, which the authors attribute to the strength of frozen foundation-model features.
- KLDA on BART-base matches or exceeds the joint fine-tuning upper bound on several text datasets without any replay data.
Limitations
- Assumes the frozen foundation model already contains features suitable for the target domain; accuracy may suffer otherwise.
- Image results still lag joint training on CIFAR100 and TinyImageNet, suggesting current vision foundation models are less expressive for complex datasets.
- Hyperparameter sigma is chosen empirically per foundation model without a validation set.
Experimental setup
Datasets: CLINC Banking DBpedia HWU CIFAR10 CIFAR100 TinyImageNet Stanford Cars
Benchmarks: CLINC Banking DBpedia HWU CIFAR10 CIFAR100 TinyImageNet Stanford Cars
Metrics: final accuracy
Baselines: Vanilla fine-tuning EWC KD L2P LAMOL VAG NCM LDA Joint Fine-tuning
Builds on: linear discriminant analysis random fourier features VAG streaming linear discriminant analysis
Closest papers by meaning
- Make Domain Shift a Catastrophic Forgetting Alleviator in Class-Incremental Learning (AAAI 2025 · similarity 0.89)
- AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection (NeurIPS 2025 · similarity 0.88)
- KAC: Kolmogorov-Arnold Classifier for Continual Learning (CVPR 2025 · similarity 0.88)
- Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings (NeurIPS 2024 · similarity 0.87)
- Defying Imbalanced Forgetting in Class Incremental Learning (AAAI 2024 · similarity 0.87)
- Class Incremental Learning via Likelihood Ratio Based Task Prediction (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.