Research map / Continual learning / Continual fine-tuning of LLMs

MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging

NeurIPS 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MINGLE, a test-time continual model merging method that uses a mixture of low-rank LoRA-style experts with per-task gates adapted on a few unlabeled test samples, and null-space constrained gating with adaptive relaxation to limit forgetting.

Problem

Continual model merging of independently fine-tuned models without training data suffers from parameter interference and forgetting, and merged models adapt poorly to shifting test distributions.

Approach

Combines OPCM-style orthogonal low-rank experts with per-task input-dependent gates that are trained at test time on a few unlabeled seed samples, constraining gate updates to the null space of previous tasks' activation subspaces and softly relaxing that constraint via an EMA of gradient-interference scores.

Key results

• On CLIP ViT-B/32 with 20 tasks, MINGLE reaches 77.1% ACC and -2.2 BWT, versus 65.7% ACC for OPCM (the strongest prior baseline).
• On ViT-B/16 with 8 tasks, MINGLE reaches 88.3% ACC and -0.4 BWT, versus 81.8% ACC for OPCM.
• On MTIL, MINGLE achieves 83.0 average last accuracy among continual merging methods, versus 77.0 for OPCM.

Measured effects reported in the paper

Surprising findings

• Adding test-time adaptation without null-space constraints raises ACC but worsens BWT substantially (e.g., -12.8 BWT at 20 tasks for ViT-B/16 ablation).
• Gate activations stay below 1.0 even without the null-space constraint, which the authors attribute to complementary, softly combined experts.

Limitations

• Code and data are not released at submission; authors promise release upon acceptance.
• Relaxed projector allows more residual interference than hard null-space projection.
• Evaluation is limited to image classification (CLIP) plus a single GLUE setup with Flan-T5-base.

Experimental setup

Datasets: SUN397 Stanford Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD Flowers102 PCAM FER2013 Oxford-IIIT Pet

Benchmarks: FusionBench CLIP 8/14/20-task merging MTIL GLUE 8-task (Flan-T5-base) corruption robustness (ImageNet-C style, 7 corruptions on 4-task ViT-B/32)

Metrics: ACC BWT

Baselines: Continual Task Arithmetic Continual Ties-Merging OPCM MagMax-Ind Consensus TA SWA C. LW AdaMerging C. LoRA-WEMOE WiSE-FT ZSCL MoE-Adapter DIKI

Builds on: OPCM Task Arithmetic WEMoE LoRA AdaMerging

Closest papers by meaning

• Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.90)
• BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.90)
• Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.89)
• Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
• When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation (NeurIPS 2026 · similarity 0.87)
• Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.87)