Research map / Continual learning / Class-incremental
Generative Multi-modal Models are Good Class Incremental Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces discriminative classifiers in class-incremental learning with a fine-tuned generative multi-modal model (MiniGPT-4 style) that generates label text, classified by nearest CLIP text-feature match to seen class names.
Problem
Classifier bias toward the current task causes catastrophic forgetting in class-incremental learning (CIL) with discriminative models that need expanding heads per task.
Approach
Use a generative multi-modal model to directly generate a class label sentence ("This is a photo of [CLS]"), then map generated text to the nearest seen class name via cosine similarity of CLIP text features, so no expanding classifier head is needed; only the linear projection layer is updated.
Key results
- On ImageNet-R 10-task conventional CIL, GMM reaches 89.41 last-task accuracy with exemplars versus 73.88 for CODA-Prompt (no exemplars).
- On mini-ImageNet few-shot CIL, GMM achieves 75.18% final-session accuracy, over 14 points above CODA-Prompt (61.14%).
- Without exemplars on Tiny-ImageNet 20-task, GMM gives 76.03 last accuracy vs 66.32 for CODA-Prompt on 10 tasks (different settings noted in table); CIFAR100 B0-20 outperforms CODA-Prompt by about 10 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GMM instead of CODA-Prompt | Tiny-ImageNet CIL · Tiny-ImageNet, 20 tasks | Last accuracy: 66.32 -> 76.03 | Helped |
| use GMM instead of CODA-Prompt | ImageNet-R CIL · ImageNet-R, 10 tasks | Last accuracy: 73.88 -> 80.72 | Helped |
| use GMM instead of CODA-Prompt | mini-ImageNet few-shot CIL · mini-ImageNet, few-shot CIL phase 8 | accuracy: 61.14 -> 75.18 | Helped |
| use GMM instead of CODA-Prompt | CIFAR100 few-shot CIL · CIFAR100, few-shot CIL phase 8 | accuracy: 59.32 -> 81.47 | Helped |
| use GMM instead of CODA-Prompt | mini-ImageNet few-shot CIL · mini-ImageNet, few-shot CIL phase 8 | Performance Drop (PD): 34.23 -> 14.17 | Helped |
| use GMM instead of CODA-Prompt | CIFAR100 few-shot CIL · CIFAR100, few-shot CIL phase 8 | Performance Drop (PD): 34.23 -> 10.06 | Helped |
| use GMM rather than discriminative prompt methods in long task sequences | conventional CIL · CIFAR100 B0-20 tasks | last-task accuracy: +10 points | Helped |
| use GMM rather than discriminative prompt methods in early tasks and short sequences | conventional CIL · CIFAR100 and Tiny-ImageNet, initial tasks and 5-task settings | last-task accuracy | Hurt |
Surprising findings
- Zero-shot frozen MiniGPT-4 with text matching already beats many traditional CIL baselines, suggesting generative models are strong continual learners even without training.
- Without exemplars, the method is slightly below DualPrompt and CODA-Prompt on short sequences (B100-5), which the authors attribute to ImageNet-21K pretraining overlap.
Limitations
- Authors acknowledge the overall design is simple and they are the first to apply this idea, so the contribution is limited in scope.
- Lower accuracy on early tasks and short sequences compared with pretrained prompt-based methods.
- Zero-shot generated text is inconsistent and verbose without fine-tuning.
Experimental setup
Datasets: CIFAR100 Tiny-ImageNet ImageNet-R mini-ImageNet
Benchmarks: CIFAR100 Tiny-ImageNet ImageNet-R mini-ImageNet
Metrics: average accuracy last-task accuracy performance drop (PD)
Baselines: EWC LwF iCaRL EEIL UCIR PASS DyTox Continual-CLIP L2P DualPrompt CODA-Prompt Linear Probe
Builds on: MiniGPT-4 BLIP-2 PILOT CLIP
Closest papers by meaning
- Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings (NeurIPS 2024 · similarity 0.89)
- Knowledge Graph Enhanced Generative Multi-modal Models for Class-Incremental Learning (NeurIPS 2025 · similarity 0.88)
- Make Domain Shift a Catastrophic Forgetting Alleviator in Class-Incremental Learning (AAAI 2025 · similarity 0.87)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.86)
- eTag: Class-Incremental Learning via Embedding Distillation and Task-Oriented Generation (AAAI 2024 · similarity 0.86)
- Long-Tail Class Incremental Learning via Independent Sub-prototype Construction (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.