Research map / Continual learning / Class-incremental
Knowledge Graph Enhanced Generative Multi-modal Models for Class-Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KG-GMM, which builds an evolving ConceptNet-based knowledge graph during class-incremental learning, trains a generative MLLM (MiniGPT-4 based) to output relation triplets, and uses relation keywords at inference to locate the class, improving exemplar-free CIL.
Problem
Exemplar-free class-incremental learning with generative multi-modal models suffers forgetting of fine-grained details of earlier classes, causing misclassification (e.g., Granny Smith predicted as Pineapple or generic "apple").
Approach
Assign each class distinct ConceptNet relation triplets as training targets so the frozen MLLM learns relational descriptions, and at inference match relation keywords in the generated text against the incrementally built subgraph to pick the head entity, then prepend it to the text for final class matching.
Key results
- On ImageNet-R B0-10 tasks, KG-GMM reaches 84.29% last accuracy vs. 80.72% for GMM and 73.88% for CODA-Prompt.
- On mini-ImageNet few-shot CIL final session, KG-GMM achieves 78.07% vs. 75.18% for GMM.
- On CIFAR-100 few-shot CIL, KG-GMM gains 4.25 points in harmonic accuracy over GMM (79.68 vs. 75.43).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the evolving knowledge graph and graph-augmented inference | GMM · Tiny-ImageNet B100-5 tasks | Last accuracy: +4.88% | Helped |
| add the evolving knowledge graph and graph-augmented inference | GMM · Tiny-ImageNet B100-10 tasks | Last accuracy | Helped |
| add the evolving knowledge graph and graph-augmented inference | GMM · Tiny-ImageNet B100-20 tasks | Last accuracy | Helped |
| add the evolving knowledge graph and graph-augmented inference | GMM · ImageNet-R B0-10 tasks | Last accuracy: 3.57% | Helped |
| add the evolving knowledge graph and graph-augmented inference | GMM · Mini-ImageNet few-shot CIL session 8 | Accuracy: 2.89 points | Helped |
| add the evolving knowledge graph and graph-augmented inference | GMM · CIFAR-100 few-shot CIL session 8 | Accuracy: 0.90% points | Helped |
| add graph-augmented inference using graph relations and generated descriptions (ablation) | KG-GMM · ImageNet-R B0-10 tasks | Accuracy: 82.77% -> 84.29% | Helped |
| replace simple text labels with GPT-3.5-generated class descriptions, without relation keywords (ablation) | GMM · ImageNet-R B0-10 tasks | Accuracy: 80.72 -> 80.95 | No clear effect |
| use graph-based prediction as the final prediction without combining it with generated text (ablation) | KG-GMM · ImageNet-R B0-10 tasks | Accuracy: 80.72 -> 82.77 | Helped |
| increase stored relations per class from r=0 to r=3 (ablation) | KG-GMM · ImageNet-R B0-10 tasks | Accuracy: 81.03 -> 84.29 | Helped |
| increase stored relations per class from r=3 to r=4 (ablation) | KG-GMM · ImageNet-R B0-10 tasks | Accuracy: 84.29 -> 84.31 | No clear effect |
Surprising findings
- Simply using GPT-3.5 generated class descriptions as text labels yields almost no improvement over GMM (80.95 vs 80.72), so relation keywords are essential.
Limitations
- Inference cost grows with longer generated text (more relations per class).
- Relies on ConceptNet coverage of class names; second-hop relations are needed when first-hop tails are taken.
Experimental setup
Datasets: ImageNet-R Tiny-ImageNet mini-ImageNet CIFAR-100 ConceptNet
Benchmarks: Tiny-ImageNet ImageNet-R mini-ImageNet CIFAR-100
Metrics: Avg accuracy Last accuracy Harmonic accuracy performance drop
Baselines: GMM Continual-CLIP L2P DualPrompt CODA-Prompt MoE-CLIP RAPF iCaRL EEIL UCIR PASS DyTox
Builds on: GMM ZSL-KG MiniGPT-4 ConceptNet SLCA
Closest papers by meaning
- Generative Multi-modal Models are Good Class Incremental Learners (CVPR 2024 · similarity 0.88)
- Towards Continual Knowledge Graph Embedding via Incremental Distillation (AAAI 2024 · similarity 0.88)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.87)
- GraphKeeper: Graph Domain-Incremental Learning via Knowledge Disentanglement and Preservation (NeurIPS 2025 · similarity 0.87)
- Toward a Unified Statistical Theory of Unsupervised Pretraining and Supervised Neural Knowledge Graph Learning (NeurIPS 2026 · similarity 0.86)
- Exploring Rationale Learning for Continual Graph Learning (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.