Research map / Continual learning / Continual fine-tuning of LLMs
Efficient Bilevel Optimization for CKA-Guided MoE Upcycling
Zhiyuan Yu, Enneng Yang, Hao Jiang, Guojie Zhu, Feihong He, Peng Wang, Li Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CKA-guided selective bilevel expansion to upcycling | Standard Upcycle · TRACE | Last Acc: 41.49 -> 45.05 | Improved |
| add CKA-guided selective bilevel expansion to upcycling | Standard Upcycle · TRACE | BWT: -8.64 -> -4.05 | Improved |
| add selective expansion instead of expanding all layers | Standard Upcycle · TRACE | model size: 6.07B -> 2.46B | Improved |
| upcycle into MoE instead of dense fine-tuning | Dense baseline · TRACE | average performance: 31.09 -> 41.21 | Improved |
| upcycle into MoE instead of dense fine-tuning | Dense baseline · TRACE | BWT: -20.13 -> -9.05 | Improved |
| add CKA regularization term (ablation) | bilevel upcycling · TRACE | forgetting | Improved |
| add knowledge-gain term in outer loop (ablation) | bilevel upcycling · TRACE | architecture selection | Improved |
| add replay-free selective upcycling vs Replay baseline | Replay · TRACE | Last Acc: 43.76 -> 45.05 | Improved |
| use Ours instead of BTM | BTM · TRACE | Last Acc: 42.50 -> 45.05 | Improved |
Closest papers by meaning
- Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization (ICLR 2025 · similarity 0.89)
- Theory on Mixture-of-Experts in Continual Learning (ICLR 2025 · similarity 0.86)
- Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025 · similarity 0.86)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.84)
- DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · similarity 0.84)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
