Research map / Continual learning / Domain and prompt-based
Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DUCT, which consolidates a pre-trained ViT's representation by merging task vectors across domains and aligns classifiers via semantic optimal-transport transport of old-domain classifiers, for exemplar-free domain-incremental learning.
Problem
Domain-incremental learning with pre-trained models suffers catastrophic forgetting in both the embedding (representation) and the classifier as sequential updates overwrite knowledge from earlier domains.
Approach
Incrementally merges domain-specific task vectors (weighted by class-center task similarity) into the pre-trained backbone to form a unified embedding, then estimates old-domain classifier weights in the consolidated space by transporting the new classifier via Sinkhorn optimal transport with pre-trained class-center distances as cost, and interpolates with the historical classifiers.
Key results
- On CDDB with ViT-B/16 IN1K, DUCT reaches 84.14% average accuracy and 85.10% final accuracy over five task orders, versus 78.92/80.48 for RanPAC and 68.51/72.76 for S-iPrompt.
- DUCT achieves the lowest forgetting measure (0.12) on CDDB among compared methods, versus 0.21 for DualPrompt.
- Ablation on CDDB shows old classifier transport adds about 2% final accuracy over the variant with representation consolidation and new classifier retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DUCT instead of CODA-Prompt | ViT-B/16 IN1K · Office-Home | average accuracy: 84.70±2.94 -> 86.27±2.95 | Helped |
| use DUCT instead of CODA-Prompt | ViT-B/16 IN1K · Office-Home | last accuracy: 85.07±0.34 -> 86.91±0.06 | Helped |
| use DUCT instead of Replay | ViT-B/16 IN1K · DomainNet | average accuracy: 64.78±2.98 -> 67.16±3.75 | Helped |
| use DUCT instead of Replay | ViT-B/16 IN1K · DomainNet | last accuracy: 61.16±1.19 -> 67.01±1.35 | Helped |
| use DUCT instead of CODA-Prompt | ViT-B/16 IN1K · CORe50 | average accuracy: 87.92±0.41 -> 91.95±0.15 | Helped |
| use DUCT instead of Replay | ViT-B/16 IN1K · CORe50 | last accuracy: 92.21±0.63 -> 94.47±0.33 | Helped |
| use DUCT instead of RanPAC | ViT-B/16 IN1K · CDDB | average accuracy: 78.92±4.85 -> 84.14±4.37 | Helped |
| use DUCT instead of RanPAC | ViT-B/16 IN1K · CDDB | last accuracy: 80.48±0.68 -> 85.10±0.52 | Helped |
| add representation consolidation by merging backbone task vectors (ablation) | Baseline on CDDB · CDDB | average accuracy: 66.56 -> 80.36 | Helped |
| use task similarity in backbone consolidation (ablation) | Variation 1 on CDDB · CDDB | average accuracy: 80.36 -> 81.41 | Helped |
| retrain the new classifier on consolidated features (ablation) | Variation 2 on CDDB · CDDB | average accuracy: 81.41 -> 85.42 | Helped |
| transport and consolidate old classifiers with semantic-guided optimal transport (ablation) | Variation 3 on CDDB · CDDB | average accuracy: 85.42 -> 87.74 | Helped |
Surprising findings
- Exemplar-based methods such as MEMO show very high forgetting (18.63) despite replay-style design, suggesting they are unsuitable for DIL.
Limitations
- Relies on generalizable pre-trained initialization; not evaluated on non-PTM settings.
- Merge hyperparameters (alpha) require tuning; very small or large values perform poorly.
Experimental setup
Datasets: Office-Home DomainNet CORe50 CDDB-Hard
Benchmarks: Office-Home DomainNet CORe50 CDDB-Hard
Metrics: average incremental accuracy last-stage accuracy forgetting measure
Baselines: Finetune SimpleCIL RanPAC EASE L2P S-iPrompt DualPrompt CODA-Prompt Replay iCaRL MEMO
Builds on: L2P DualPrompt CODA-Prompt task arithmetic model merging Sinkhorn optimal transport
Closest papers by meaning
- Gradual Divergence for Seamless Adaptation: A Novel Domain Incremental Learning Method (ICML 2024 · similarity 0.87)
- GraphKeeper: Graph Domain-Incremental Learning via Knowledge Disentanglement and Preservation (NeurIPS 2025 · similarity 0.86)
- Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Experts (ICML 2025 · similarity 0.85)
- DualCP: Rehearsal-Free Domain-Incremental Learning via Dual-Level Concept Prototype (AAAI 2025 · similarity 0.85)
- Tripartite Weight-Space Ensemble for Few-Shot Class-Incremental Learning (CVPR 2025 · similarity 0.85)
- Componential Prompt-Knowledge Alignment for Domain Incremental Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.