Research map / Continual learning / Online and replay-based
Confounder-Free Continual Learning via Recursive Feature Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Recursive MDN (R-MDN), a normalization layer that uses recursive least squares to residualize confounder effects from DNN features online, enabling confounder-free learning in continual and static settings.
Problem
Removing confounder influence from deep network features is hard in continual learning, where data arrive sequentially and batch-level statistics needed by MDN are unavailable; batch-statistic methods also do not suit vision transformers.
Approach
Replace MDN's batch-level closed-form OLS with a recursive least squares update (Sherman-Morrison / Woodbury) that maintains an internal regression state (coefficients and inverse covariance) updated per example or mini-batch, so the layer can be inserted anywhere, including in vision transformers, and adapt to changing confounder distributions.
Key results
- On the synthetic continual dataset 3, R-MDN reaches ACCd 0.02 versus 0.28 for the CNN baseline and 0.04 for P-MDN, with lowest FWTd.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply R-MDN to CNN features | CNN continual learning · Synthetic continual Dataset 3 (both distributions change) | ACCd: 0.28 ± 0.00 -> 0.02 ± 0.01 | Helped |
| Apply R-MDN to ViT | 2D ViT · Synthetic continual Dataset 3 | ACCd: 0.110 ± 0.027 -> 0.061 ± 0.013 | Helped |
| Apply R-MDN to HAM10000 ViT | ViT continual skin lesion classification · HAM10000 | Average dcor^2: 0.0864 ± 0.0336 -> 0.0475 ± 0.0247 | Helped |
| Use R-MDN only after the pre-logits layer rather than after attention or every transformer block (ablation) | R-MDN on HAM10000 ViT · HAM10000 | Accuracy: 0.5503 ± 0.0541 -> 0.6919 ± 0.0723; 0.5288 ± 0.0571 -> 0.6919 ± 0.0723 | Helped |
| Add R-MDN (C) to LwF (ablation) | LwF on HAM10000 · HAM10000 | Average dcor^2: 0.0512 ± 0.0407 -> 0.0354 ± 0.0210 | Helped |
| Add R-MDN (C) to EWC (ablation) | EWC on HAM10000 · HAM10000 | Average dcor^2: 0.0938 ± 0.0506 -> 0.0592 ± 0.0488 | Helped |
| Apply R-MDN to static ABCD sex classification | CNN · ABCD sex classification | TPR - TNR: -0.029 ± 0.016 -> -0.002 ± 0.018 | Helped |
| Apply R-MDN to static synthetic learning | CNN, batch size 2 · Synthetic static learning | |bAcc - A|: 10.49 ± 0.037 -> 0.28 ± 0.414 | Helped |
| Vary R-MDN placement from after self-attention in every block (A) to only after pre-logits (C) (ablation) | R-MDN on HAM10000 ViT · HAM10000 | Accuracy: 0.5503 ± 0.0541 -> 0.6919 ± 0.0723 | Helped |
| Vary R-MDN placement from every transformer-block end (B) to only after pre-logits (C) (ablation) | R-MDN on HAM10000 ViT · HAM10000 | Accuracy: 0.5288 ± 0.0571 -> 0.6919 ± 0.0723 | Helped |
Surprising findings
- R-MDN trades away some classification accuracy on HAM10000 and ABCD in exchange for reduced confounder dependence, so higher accuracy can reflect confounder cheating.
- Analytic continual learning methods ACIL and F-OAL show near-zero BWTd but much worse ACCd and FWTd on the synthetic data, indicating they exploit confounders.
Limitations
- Q-matrix is not recomputed during continual learning and is estimated instead of exact.
- Accuracy drops relative to baselines on some real datasets (HAM10000 and ABCD).
- Experiments are limited to small CNNs and ViTs on medical imaging with a single confounder type per study.
Experimental setup
Datasets: HAM10000 ADNI ABCD synthetic continual datasets
Benchmarks: HAM10000 ADNI ABCD synthetic continual datasets
Metrics: ACCd BWTd FWTd accuracy dcor^2 TPR-TNR difference BWT FWT
Baselines: ViT Baseline CNN Baseline BR-Net MDN P-MDN EWC LwF PackNet ACIL F-OAL Pixel-Space MDN
Builds on: metadata normalization P-MDN BR-Net Sherman-Morrison formula recursive least squares
Closest papers by meaning
- Where is the Truth? The Risk of Getting Confounded in a Continual World (ICML 2025 · similarity 0.88)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.87)
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.86)
- A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · similarity 0.85)
- Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.85)
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.