Research map / Continual learning / Online and replay-based
Semantic Concept Steering Breaks the Explanation Drift Loop in Continual Learning
Yehonatan Elisha, Oren Barkan, Noam Koenigstein
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a semantic concept steering approach to continual learning that addresses explanation drift, where models keep accuracy while shifting attention to spurious features, by fixing a flaw in pixel-level saliency regularization.
Problem
Continual learning models suffer explanation drift (attention shifting from diagnostic to spurious features) even when accuracy is maintained; existing pixel-level saliency regularization derives its supervision targets from the model's own drifting saliency maps.
Approach
Identifies a structural flaw in explanation-aware CL: pixel-level saliency regularization takes supervision targets from the model's own (already drifted) maps. Proposes replacing these with semantic concept-level steering to break the drift loop. Details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace pixel-level saliency targets with concept-level targets (C3L with AttnLRP vs RRR-ViT-AttnLRP) | RRR-ViT-AttnLRP · CUB (ViT-B) | ACC: 78.09 -> 83.65 | Improved |
| add concept overlap loss L_concept (ablation) | C3L · CUB (ViT-B) | ACC: 73.74 -> 83.65 | Improved |
| add background stability loss L_stab (ablation) | C3L · CUB (ViT-B) | ACC: 78.42 -> 83.65 | Improved |
| add concept overlap loss L_concept (ablation) | C3L · ImageNet-R (ViT-B) | ACC: 77.12 -> 82.74 | Improved |
| add background stability loss L_stab (ablation) | C3L · ImageNet-R (ViT-B) | ACC: 78.94 -> 82.74 | Improved |
| stabilize background (cosine) rather than suppress it (ablation) | C3L · ImageNet-R (ViT-B) | ACC: 78.94 -> 82.74 | Improved |
| add replay buffer to semantic regularization (full C3L vs NoReplay) (ablation) | C3L · ImageNet-R (ViT-B) | ACC: 79.82 -> 82.74 | Improved |
| apply semantic regularization without any replay (C3L-NoReplay vs ER) (ablation) | ER · ImageNet-R (ViT-B) | ACC: 72.27 -> 79.82 | Improved |
| regularize replay with saved model saliency targets (FullReplay) instead of concept targets (ablation) | C3L · ImageNet-R (ViT-B) | ACC: 82.74 -> 80.32 | Hurt |
| use AttnLRP instead of Grad-CAM as explanation method (ablation) | C3L · ImageNet-R (ViT-B) | ACC: 80.92 -> 82.74 | Improved |
| reduce replay buffer from 1000 to 200 (ablation) | C3L · ImageNet-R (ViT-B) | ACC: 82.74 -> 80.40 | Hurt |
| use LRP/AttnLRP in RRR instead of Grad-CAM | RRR-ViT · CUB (ViT-B) | ACC: 76.94 -> 78.09 | Improved |
Closest papers by meaning
- Language Guided Concept Bottleneck Models for Interpretable Continual Learning (CVPR 2025 · similarity 0.91)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.90)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.87)
- Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Disentangling and mitigating the impact of task similarity for continual learning (NeurIPS 2024 · similarity 0.87)
- Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Models (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
