Research map / Continual learning / Continual fine-tuning of LLMs
Spectral Mixture-of-Experts for Continual Learning
Chen Yin, Xingbo Dong, Xuelin Shen, Zhe Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add router consistency projection Hw (ablation) | Spectral MoE · MTIL full-shot (11 tasks) | Average accuracy (%): 75.1 -> 78.1 | Improved |
| add dynamic expert projection He(Dη) (ablation) | Spectral MoE · MTIL full-shot (11 tasks) | Last accuracy (%): 79.8 -> 86.3 | Improved |
| add both DCP components on top of Spectral Experts (ablation) | Spectral MoE · MTIL full-shot (11 tasks) | Average accuracy (%): 75.0 -> 78.1 | Improved |
| replace LoRA experts with Spectral Experts (ablation) | Spectral MoE · Forgetting AUC, full-shot | Forgetting AUC (%): 1.81% -> 0.91% | Improved |
| replace LoRA experts with Spectral Experts (ablation) | Spectral MoE · Forgetting AUC, few-shot | Forgetting AUC (%): 1.68% -> 0.61% | Improved |
| use full stability Dη = I instead of dynamic Dη (ablation) | Spectral MoE · MTIL full-shot (11 tasks) | Transfer accuracy (%): 70.1 -> 68.7 | Hurt |
| use full plasticity Dη = 0 instead of dynamic Dη (ablation) | Spectral MoE · MTIL full-shot (11 tasks) | Last accuracy (%): 86.3 -> 80.3 | Hurt |
| set stability floor eta_min to 0 instead of 0.95 (ablation) | Spectral MoE · MTIL full-shot (11 tasks) | accuracy | Hurt |
| use focusing strength gamma 1 or >=3 instead of 2 (ablation) | Spectral MoE · MTIL full-shot (11 tasks) | accuracy | Hurt |
| increase number of experts up to N=32 (ablation) | Spectral MoE · MTIL full-shot (11 tasks) | accuracy | Improved |
| use Spectral Experts instead of LoRA experts in MoE adapter | MoE adapters · Training cost | Trainable parameters: 59.8M -> 23.5 M | Improved |
| use Spectral MoE instead of MoE-Adapters++ | CLIP continual learning · MTIL full-shot (11 tasks) | Average accuracy (%): 77.5 -> 78.1 | Improved |
Closest papers by meaning
- Theory on Mixture-of-Experts in Continual Learning (ICLR 2025 · similarity 0.91)
- Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning (AAAI 2025 · similarity 0.88)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.88)
- Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation (CVPR 2024 · similarity 0.87)
- BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · similarity 0.87)
- Mixture of Tokens: Continuous MoE through Cross-Example Aggregation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
