Research map / Large language models / Instruction and fine-tuning

Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SCF-RKL, a data-free model merging method that uses reverse-KL-weighted parameter deviation with an IQR threshold to sparsely copy parameters from a second model, aiming to avoid generation degradation (repetition) seen in dense merging.

Problem

Parameter-space merging methods (Task Arithmetic, TIES, DARE, SCE) can amplify repetition and degrade generation stability and capabilities when composing LLMs.

Approach

Score each parameter by |theta_s - theta_b| times a reverse KL between softmax-distributions over parameter groups, then keep only high-score parameters (above median + 1.5 IQR) from the complementary model, yielding a binary-mask sparse fusion.

Key results

• On TinyR1-32B fusion, SCF-RKL reaches average pass@1 65.33 vs 64.47 for the stronger base model and 57.80-58.22 for baselines, with near-100% repetition for baselines on GSM8K versus ~0.2% for SCF-RKL (as reported).
• On Mistral-7B fusion, SCF-RKL averages 43.70 but DARE-TIES averages 43.97 (higher), so SCF-RKL does not win on that table; on Llama-3-8B it averages 53.01 vs 52.60 for Task Arithmetic.
• On seven vision classification tasks with ViT-B/32, SCF-RKL averages 63.94% vs 62.29% for Task Arithmetic and 45.89% for SCE.

Surprising findings

• Dense merging baselines (Task Arithmetic, DARE, SCE) collapse to near-zero GSM8K scores on several Llama-3 and Mistral fusions, showing severe generation failures even when base models are fine.
• The paper's theorems are informal: the entropy bound and reverse-KL monotonicity proofs rely on loose assumptions and the mask is applied to raw parameters interpreted via softmax, which is an unusual choice.

Limitations

• Theoretical guarantees are sketched with loose assumptions (e.g., treating raw weights as softmax logits; Lipschitz constants unspecified), so the bounds are not rigorous.
• Results are mixed: on Mistral-7B SCF-RKL is below DARE-TIES on average, and on safety/instruction tables gains are small or inconsistent.
• Evaluation relies on single-run pass@1 numbers without variance estimates, and the reverse-KL score is computed over arbitrary parameter groups.

Experimental setup

Datasets: GSM8K BBH MMLU HumanEval IFEval IFBench SUN397 Cars RESISC45 DTD GTSRB MNIST

Benchmarks: AIME24 AIME25 GPQA LiveCodeBench v5 GSM8K BBH MMLU HumanEval IFEval IFBench S-Eval HarmBench

Metrics: pass@1 pass@n repetition rate accuracy normalized spectral shift

Baselines: Task Arithmetic DARE-Task Arithmetic TIES-Merging DARE-TIES SCE

Builds on: Task Arithmetic TIES-Merging DARE SCE

Closest papers by meaning

• InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion (NeurIPS 2025 · similarity 0.89)
• MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.88)
• Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.88)
• Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.87)
• Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.87)
• Mixture-Trained Merging for Unified Multi-Objective Models (NeurIPS 2026 · similarity 0.87)