Research map / Continual learning / Continual fine-tuning of LLMs
Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture of Noise (Min), which learns task-specific positive-incentive noise generators inserted between ViT blocks and mixes them with learnable weights for exemplar-free pre-trained-model class-incremental learning.
Problem
Class-incremental learning with pre-trained models suffers from parameter drift that degrades the backbone's generalization; the paper asks whether beneficial (positive-incentive) noise can counter this while keeping the backbone frozen.
Approach
Model parameter drift as harmful noise and learn task-specific positive-incentive noise generators (Pi-Noise layers between transformer blocks) on top of an analytic classifier, then mix the noises with weights set by task-prototype similarity and refined through an auxiliary classifier, so a single backbone pass suffices at inference.
Key results
- On CIFAR-100 with ViT-B/16-IN21K at 50 steps, Min reaches 93.63% average and 89.82% last accuracy, versus the runner-up at 92.36%/87.39%.
- On ImageNet-A with ViT-B/16-IN21K, Min improves last-task accuracy by roughly 6-9 points over the runner-up across settings, per the paper's text.
- Ablation on the 10-step setting shows the noise expansion and noise mixture components each contribute; MIN achieves 92.12% last accuracy on CIFAR-100 versus 86.82% for the analytic baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MiN's task-specific noise generation and adaptive noise mixture | Analytic classifier with ViT-B/16-IN21K · CIFAR-100, 50 steps | average accuracy: 92.36 {${\pm 0.04}$} -> 93.63 {${\pm 0.20}$} | Helped |
| Add MiN's task-specific noise generation and adaptive noise mixture | Analytic classifier with ViT-B/16-IN21K · CUB-200, 50 steps | average accuracy: 91.99 {${\pm 0.12}$} -> 93.21 {${\pm 0.14}$} | Helped |
| Add MiN's task-specific noise generation and adaptive noise mixture | Analytic classifier with ViT-B/16-IN21K · ImageNet-A, 50 steps | average accuracy: 63.72 {${\pm 0.65}$} -> 66.15 {${\pm 0.35}$} | Helped |
| Add MiN's task-specific noise generation and adaptive noise mixture | Analytic classifier with ViT-B/16-IN21K · ImageNet-R, 50 steps | average accuracy: 79.35 {${\pm 0.31}$} -> 82.26 {${\pm 0.16}$} | Helped |
| Add MiN's task-specific noise generation and adaptive noise mixture | Analytic classifier with ViT-B/16-IN21K · FOOD-101, 50 steps | average accuracy: 91.57 {${\pm 0.10}$} -> 93.60 {${\pm 0.07}$} | Helped |
| Add MiN's task-specific noise generation and adaptive noise mixture | Analytic classifier with ViT-B/16-IN21K · Omnibenchmark, 50 steps | average accuracy: 85.06 {${\pm 0.12}$} -> 87.58 {${\pm 0.15}$} | Helped |
| Add task-specific noise generation and noise mixing (ablation) | Analytic classifier with ViT-B/16-IN21K · CIFAR-100, 10 steps | average accuracy: 91.18 {${\pm 0.03}$} -> 95.12 {${\pm 0.05}$} | Helped |
| Use only mean components of task noise (ablation) | MiN · CIFAR-100, 10 steps | average accuracy: 95.12 {${\pm 0.05}$} -> 89.27 {${\pm 0.24}$} | Hurt |
| Use only variance components of task noise (ablation) | MiN · CIFAR-100, 10 steps | average accuracy: 95.12 {${\pm 0.05}$} -> 94.37 {${\pm 0.05}$} | Hurt |
| Use only the last task's noise (ablation) | MiN · CIFAR-100, 10 steps | average accuracy: 95.12 {${\pm 0.05}$} -> 92.36 {${\pm 0.18}$} | Hurt |
| Use noise from a random task (ablation) | MiN · CIFAR-100, 10 steps | average accuracy: 95.12 {${\pm 0.05}$} -> 92.65 {${\pm 0.21}$} | Hurt |
Surprising findings
- Injecting learned noise into intermediate features, which is usually considered harmful, improves class-incremental accuracy, supporting the claim that some noise is beneficial.
Limitations
- Depends on a strong pre-trained feature extractor to guide the Pi-Noise generators.
- Adds extra down/up projection parameters (about 4% of ViT-B/16) in each Pi-Noise layer, though the trainable count is small.
Experimental setup
Datasets: CIFAR-100 CUB-200 ImageNet-A ImageNet-R FOOD-101 Omnibenchmark
Benchmarks: CIFAR-100 CUB-200 ImageNet-A ImageNet-R FOOD-101 Omnibenchmark
Metrics: average incremental accuracy last-task accuracy
Baselines: L2P DualPrompt CODA-Prompt SLCA FeCAM RanPAC APER EASE COFiMA MOS ACIL
Builds on: ACIL Positive-incentive noise L2P SLCA
Closest papers by meaning
- Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning (ICML 2025 · similarity 0.87)
- Compress to One Point: Neural Collapse for Pre-Trained Model-Based Class-Incremental Learning (AAAI 2025 · similarity 0.87)
- Knowledge Memorization and Rumination for Pre-trained Model-based Class-Incremental Learning (CVPR 2025 · similarity 0.86)
- Prediction Error-based Classification for Class-Incremental Learning (ICLR 2024 · similarity 0.86)
- Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning (NeurIPS 2024 · similarity 0.86)
- Long-Tail Class Incremental Learning via Independent Sub-prototype Construction (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.