Research map / Continual learning / Continual fine-tuning of LLMs
Enhancing Visual Continual Learning with Language-Guided Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LingoCL, which replaces the randomly initialized trainable classifier in continual learning with frozen semantic targets generated from category names by a pretrained text encoder (CLIP), and plugs it into eleven CL baselines.
Problem
Catastrophic forgetting in continual learning: standard CL methods use one-hot labels and randomly initialized trainable classifier heads, which causes representation drifting and inefficient knowledge transfer across tasks.
Approach
Generate classifier weights once from category names using a pretrained language model (CLIP text transformer), keep them frozen during CL training so they act as semantic supervision targets that encode correlations among all classes, and integrate this into existing CL methods with minimal changes.
Key results
- On ImageNet-100 class-incremental (B=50, C=10 and related settings), LingoCL improves LUCIR-based average accuracy by 3.2% to 6.1% and reduces forgetting by 2.6% to 13.1%.
- On 10-split CIFAR-100 task-incremental learning, LingoCL improves last accuracy by 3.9% to 9.7% across regularization baselines.
- In the ablation, a frozen PLM-initialized classifier reaches 62.2% average accuracy versus 57.1% for a randomly initialized trainable classifier (LUCIR, ImageNet-100 B=10, C=10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LingoCL language-guided supervision | LUCIR · ImageNet-100 class-IL (B=50, C=10) | Avg accuracy (%): 70.2±0.0 -> 73.4±1.1 | Helped |
| Add LingoCL language-guided supervision | LUCIR · ImageNet-100 class-IL (B=50, C=5) | Avg accuracy (%): 67.7±0.8 -> 71.5±0.6 | Helped |
| Add LingoCL language-guided supervision | LUCIR · ImageNet-100 class-IL (B=10, C=10) | Avg accuracy (%): 57.1±0.8 -> 62.2±0.9 | Helped |
| Add LingoCL language-guided supervision | LUCIR · ImageNet-100 class-IL (B=5, C=5) | Avg accuracy (%): 47.5±0.5 -> 53.6±1.0 | Helped |
| Add LingoCL language-guided supervision | LUCIR · CIFAR100 class-IL (B=10, C=10) | Avg accuracy (%): 60.2±0.4 -> 61.9±0.3 | Helped |
| Add LingoCL language-guided supervision | SI · CIFAR100 task-IL | Last accuracy (%): 41.4±1.7 -> 51.1±1.5 | Helped |
| Add LingoCL language-guided supervision | MAS · OfficeHome domain-IL | Last accuracy (%): 54.1±1.3 -> 58.1±2.0 | Helped |
| Initialize the classifier with PLM weights and keep them trainable, rather than freezing them (ablation) | LUCIR · ImageNet-100 class-IL (B=10, C=10) | Avg accuracy (%): 57.1% -> 60.3% | Helped |
| Freeze PLM classifier weights instead of keeping them trainable (ablation) | LUCIR · ImageNet-100 class-IL (B=10, C=10) | Avg accuracy (%): 60.3% -> 62.2% | Helped |
| Preserve semantic correlations in the PLM classifier instead of orthogonalizing the targets (ablation) | LUCIR · ImageNet-100 class-IL (B=10, C=10) | Avg accuracy (%): 60.2% -> 62.2% | Helped |
| Replace the frozen oracle classifier with the frozen PLM classifier (ablation) | LUCIR · ImageNet-100 class-IL (B=10, C=10) | Avg accuracy (%): 61.6% -> 62.2% | Helped |
| Use OpenCLIP instead of CLIP as the pretrained language model (ablation) | LingoCL · ImageNet-100 class-IL (B=50, C=5) | Avg accuracy (%): 67.5% -> 68.0% | Helped |
Surprising findings
- LingoCL slightly surpasses an oracle classifier (trained on all tasks' data) in average accuracy and forgetting rate, which the authors attribute to the broader pretraining data of the language model.
- Updating the PLM-initialized classifier during training reduces accuracy (61.7% to 60.3%), so freezing is necessary.
Limitations
- Gains on task-incremental and domain-incremental settings are reported without a full comparison to other PLM choices in those settings.
- Relies on the availability of category names and a pretrained language model whose semantics cover the classes.
Experimental setup
Datasets: CIFAR-100 ImageNet-100 Office-Home
Benchmarks: CIFAR-100 ImageNet-100 Office-Home
Metrics: average incremental accuracy last-step accuracy forgetting rate
Baselines: AANet DyTox BiMeCo LUCIR BiC CwD IL2M EWC MAS GEM SI
Builds on: LUCIR BiC DyTox AANet CLIP
Closest papers by meaning
- Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning (AAAI 2026 · similarity 0.91)
- Vision-language Incremental Learning with Dual Class-individual Memory (AAAI 2026 · similarity 0.90)
- Language Guided Concept Bottleneck Models for Interpretable Continual Learning (CVPR 2025 · similarity 0.90)
- Forget Forgetting: Continual Learning in a World of Abundant Memory (ICLR 2026 · similarity 0.90)
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.89)
- Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.