Research map / Continual learning / Continual fine-tuning of LLMs
Robust and Efficient Continual Model Merging via Global Singular Subspace Separation and Restoration
Juncheol Shin, Byeongho Yu, Semin Kim, Eunhyeok Park
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GS3R, a data-free framework for continual model merging that separates and restores task updates in a global singular subspace.
Problem
Continual model merging, where task-specific models arrive sequentially, suffers from task balance collapse and substantial performance degradation when standard merging approaches are naively extended.
Approach
A data-free, optimization-compatible framework that separates and restores task contributions within a global singular subspace to balance tasks during sequential merging (details beyond the abstract not available).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add vector restoration to continual SVD merging (ablation) | GS3R · ViT-B/32, 20 tasks | accuracy: 55.6 -> 61.7 | Improved |
| add global singular subspace separation without restoration (ablation) | GS3R · ViT-B/32, 20 tasks | accuracy: 55.6 -> 55.6 | No clear effect |
| add global separation on top of restoration (ablation) | GS3R · ViT-B/32, 20 tasks | accuracy: 61.7 -> 72.7 | Improved |
| truncate restoration state to r=256 (ablation) | GS3R · ViT-B/32, 20 tasks | accuracy: 72.7 -> 70.9 | Hurt |
| add auxiliary optimization (ablation) | GS3R · ViT-B/32, 20 tasks | accuracy: 72.7 -> 73.2 | Improved |
| truncate to r=64 instead of full rank (ablation) | GS3R · GLUE, Flan-T5-Base | accuracy: 82.6 -> 83.7 | Improved |
| use GS3R instead of NUFILT | GS3R · ViT-B/32, 20 tasks | accuracy: 71.0 -> 72.7 | Improved |
| use GS3R (no optimization) instead of NUFILT | GS3R · ViT-B/32, 8 tasks | accuracy: 83.6 -> 82.7 | Hurt |
| increase tolerance to 10^-2 (ablation) | GS3R · ViT-B/32, 8 tasks | accuracy: 82.7 -> 81.6 | Hurt |
| use GS3R with optimization vs NUFILT under original scorers | GS3R + Opt. · LLaVA-1.5-7B multimodal | accuracy: 66.6 -> 70.3 | Improved |
Closest papers by meaning
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.90)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.89)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.87)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.87)
- BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · similarity 0.87)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
