AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Efficient Bilevel Optimization for CKA-Guided MoE Upcycling

Zhiyuan Yu, Enneng Yang, Hao Jiang, Guojie Zhu, Feihong He, Peng Wang, Li Shen

ICML 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Synthetic data and merging for CLIP

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add CKA-guided selective bilevel expansion to upcyclingStandard Upcycle · TRACELast Acc: 41.49 -> 45.05Improved
add CKA-guided selective bilevel expansion to upcyclingStandard Upcycle · TRACEBWT: -8.64 -> -4.05Improved
add selective expansion instead of expanding all layersStandard Upcycle · TRACEmodel size: 6.07B -> 2.46BImproved
upcycle into MoE instead of dense fine-tuningDense baseline · TRACEaverage performance: 31.09 -> 41.21Improved
upcycle into MoE instead of dense fine-tuningDense baseline · TRACEBWT: -20.13 -> -9.05Improved
add CKA regularization term (ablation)bilevel upcycling · TRACEforgettingImproved
add knowledge-gain term in outer loop (ablation)bilevel upcycling · TRACEarchitecture selectionImproved
add replay-free selective upcycling vs Replay baselineReplay · TRACELast Acc: 43.76 -> 45.05Improved
use Ours instead of BTMBTM · TRACELast Acc: 42.50 -> 45.05Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy