Research map / Continual learning / Online and replay-based
Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLDNet, a rehearsal-free, minimal-expansion continual learning method combining HSIC-bottleneck orthogonal gradient projection (HBO) with a parameter-free equiangular basis vector classifier (EAE).
Problem
Class-incremental continual learning without exemplar buffers or network expansion, where catastrophic forgetting from layer-wise parameter overwriting and decision boundary distortion harms stability-plasticity balance.
Approach
Recasts orthogonal-projection gradient updates (OWM-style) as HSIC-bottleneck dependence minimization/maximization, computed with a recursive least squares update, and replaces the trainable softmax classifier with predefined equiangular basis vectors so the output head adds no trainable parameters.
Key results
- On Split CIFAR-100 with zero buffer, CLDNet reaches 65.42% average accuracy, versus 52.67% for DualHSIC at buffer size 200 (rehearsal-based comparison).
- On Split CIFAR-100 with 1.02x capacity, CLDNet achieves 64.99% Avg. Acc, above PCL (62.58% at 1.46x).
- On Split ImageNet-R, Ours(2) with two ViTs reaches 71.43% vs. DualPrompt 68.13% with no buffer; Ours(1) reaches 65.37% vs. L2P 61.57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CLDNet instead of AOP | CLDNet · Split MNIST | average accuracy: 94.43 -> 96.61 | Helped |
| use CLDNet instead of CRNet | CLDNet · FashionMNIST | average accuracy: 90.98 -> 95.37 | Helped |
| use CLDNet instead of AOP | CLDNet · CIFAR-10 | average accuracy: 53.56 -> 56.12 | Helped |
| use rehearsal-free CLDNet instead of LiDER with buffer size 2000 | CLDNet · Split CIFAR-100 | average accuracy: 62.78$\pm$0.51 -> 65.42$\pm$0.36 | Helped |
| use two pre-trained ViT models instead of one | CLDNet · Split ImageNet-R | average accuracy: 65.37$\pm$0.39 -> 71.43$\pm$0.22 | Helped |
| use CLDNet Ours(2) instead of DualPrompt | CLDNet Ours(2) · Split ImageNet-R | average accuracy: 68.13$\pm$0.49 -> 71.43$\pm$0.22 | Helped |
| use CLDNet instead of PCL | CLDNet · Split CIFAR-100 | average accuracy: 62.58$\pm$0.32 -> 64.99$\pm$0.24 | Helped |
| replace EAE with a fully connected classifier trained without an orthogonal projector (ablation) | HBO · Split MNIST | accuracy: 96.62 -> 19.96 | Hurt |
| replace EAE with a fully connected classifier trained with an orthogonal projector (ablation) | HBO · Split MNIST | accuracy: 96.62 -> 95.12 | Hurt |
| replace EAE with a fully connected classifier trained without an orthogonal projector (ablation) | HBO · FashionMNIST | accuracy: 95.35 -> 19.74 | Hurt |
| replace EAE with a fully connected classifier trained with an orthogonal projector (ablation) | HBO · FashionMNIST | accuracy: 95.35 -> 92.77 | Hurt |
| replace EAE with a fully connected classifier trained without an orthogonal projector (ablation) | HBO · CIFAR-10 | accuracy: 56.10 -> 15.56 | Hurt |
Surprising findings
- Rehearsal-free CLDNet with zero buffer outperforms several rehearsal-based methods that use buffers of 200 to 2000 samples on Split CIFAR-100.
Limitations
- Relies on hyperparameters (beta=500, gamma=0.04, adaptive alpha) selected per setting; sensitivity not fully analyzed.
- On ImageNet-R, Ours(1) falls short of DualPrompt by 2.76%, and no buffered comparison is given at buffer size 0 for most baselines.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR-10 CIFAR-100 ImageNet-R
Benchmarks: Split MNIST Split FashionMNIST Split CIFAR-10 Split CIFAR-100 Split ImageNet-R
Metrics: average accuracy capacity (model size ratio)
Baselines: EWC SI MAS OEWC ICNet DMC OWM AOP CRNet ER DER++ X-DER-RPC
Builds on: OWM AOP DualHSIC equiangular basis vectors Hilbert-Schmidt independence criterion
Closest papers by meaning
- Adversarial Latent Embedding Repair for LLM Continual Learning (ICML 2026 · similarity 0.88)
- BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · similarity 0.87)
- Expandable and Differentiable Dual Memories with Orthogonal Regularization for Exemplar-free Continual Learning (AAAI 2026 · similarity 0.87)
- Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning (ICLR 2026 · similarity 0.87)
- Learning Equi-angular Representations for Online Continual Learning (CVPR 2024 · similarity 0.86)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.