Research map / Continual learning / Continual fine-tuning of LLMs
Walking the Web of Concept-Class Relationships in Incrementally Trained Interpretable Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MuCIL, a concept-based class-incremental learner that fuses image patches with text-anchored concept embeddings in a transformer encoder, classifying parameter-free via dot products with class text embeddings, and evaluates on CIFAR-100, CUB200 and ImageNet-100 with new concept-forgetting metrics.
Problem
Concept-based models in class-incremental learning where new classes may reuse old concepts and introduce new ones; existing methods forget concept-class relationships and grow parameters with each experience.
Approach
A multimodal image-concept transformer encoder that appends concept text embeddings to image patch tokens, grounds concept outputs to text anchors via a linear-alignment loss, and classifies with parameter-free dot products against class text embeddings, so new concepts and classes add no trainable parameters.
Key results
- On 5-experience ER with buffer 500, MuCIL reaches FAA 0.80 on ImageNet-100 and 0.78 on CUB, versus at most 0.41 and 0.42 for the best ER baselines (LaBo, Label-Free).
- CCRF on ImageNet-100 is 0.0172 for MuCIL versus 0.1450 for CBM-J, and 0.0099 vs 0.0362 on CUB.
- In the full-data single-experience setting MuCIL gets 0.8401 on CUB, versus 0.7413 for the next best baseline (Label-Free), while LaBo is best on CIFAR-100 and ImageNet-100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MuCIL instead of concept-based continual-learning baselines | CIFAR-100, CUB, and ImageNet-100 continual learning · 5 and 10 experiences, buffer size 500 | final average accuracy | Helped |
| Use MuCIL instead of CBM-J | CUB full-data classification · CUB | classification accuracy: 0.7231 -> 0.8401 | Helped |
| Add concept-label replay to experience replay (ablation) | MuCIL · CIFAR-100, CUB200, and ImageNet-100 | linear accuracy: 0.4511 -> 0.7650; 0.1382 -> 0.7914; 0.5903 -> 0.7722 | Helped |
| Add concept-label replay to experience replay (ablation) | MuCIL · CIFAR-100, CUB200, and ImageNet-100 | FAA: 0.6722 -> 0.7022; 0.7844 -> 0.8137; 0.7903 -> 0.7970 | Helped |
| Reduce the weight of the concept-presence loss (ablation) | MuCIL · CUB and ImageNet-100 | linear accuracy | Hurt |
| Reduce the weight of the concept-grounding loss (ablation) | MuCIL · CUB and ImageNet-100 | concept-anchor similarity | Hurt |
| Vary λ1 and λ2 (ablation) | MuCIL · CUB and ImageNet-100 | FAA | No clear effect |
| Replace vanilla attention with linear attention (ablation) | MuCIL · CIFAR-100, CUB, and ImageNet-100, 5 experiences | FAA: 0.70 -> 0.69; 0.81 -> 0.82; 0.80 -> 0.80 | No clear effect |
| Use MuCIL rather than CBM-J | CIFAR-100, CUB, and ImageNet-100 · 5 experiences, buffer size 2000 | CCRF: 0.0929 -> 0.0444; 0.0362 -> 0.0099; 0.1450 -> 0.0172 | Helped |
| Use a ViT vision encoder with a FLAVA text encoder instead of pre-aligned encoder pairs (ablation) | MuCIL · CIFAR-100, ImageNet-100, and CUB200 | FAA: 0.712; 0.792; 0.822 | Helped |
| Increase replay buffer size from 500 to 2000 or 5000 (ablation) | MuCIL with ER · 5 experiences on CIFAR-100, CUB, and ImageNet-100 | FAA: 0.67 -> 0.71 -> 0.74; 0.78 -> 0.82 -> 0.83; 0.80 -> 0.79 -> 0.81 | Helped |
Surprising findings
- Pre-aligned vision-language encoders (CLIP, FLAVA) performed worse than dedicated unimodal encoders (ViT with FLAVA text) for this method.
Limitations
- Forgetting bias toward the concept set of the final experience remains.
- Relies on replay buffers; authors note CL methods that do not replay labels are an open direction.
- Concepts for CIFAR-100 and ImageNet-100 are noisy LLM-generated labels.
Experimental setup
Datasets: CIFAR-100 CUB200 ImageNet-100
Benchmarks: CIFAR-100 CUB200 ImageNet-100
Metrics: Final Average Accuracy Average Forgetting Concept-Class Relationship Forgetting Concept Linear Accuracy Active Concept Ratio accuracy
Baselines: CBM-J CBM-S ICIAP-J ICIAP-S Label-Free CBM LaBo ICICLE
Builds on: Concept Bottleneck Models Experience Replay DER++ A-GEM FLAVA
Closest papers by meaning
- A Concept-Based Explainability Framework for Large Multimodal Models (NeurIPS 2024 · similarity 0.86)
- Language Guided Concept Bottleneck Models for Interpretable Continual Learning (CVPR 2025 · similarity 0.86)
- Restyling Unsupervised Concept Based Interpretable Networks with Generative Models (ICLR 2025 · similarity 0.86)
- Understanding Inter-Concept Relationships in Concept-Based Models (ICML 2024 · similarity 0.86)
- Cascaded Sparse Autoencoders LearnMulti-Level Visual Concepts in Multimodal LLMs (NeurIPS 2026 · similarity 0.86)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.