Research map / Architectures and efficiency / RNNs and normalization

Rethinking Layer-wise Model Merging through Chain of Merges

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Representation geometry and collapse · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Chain of Merges (CoM), a layer-wise model merging method that recursively recomputes activation Gram matrices from already-merged earlier layers, with correlation-based task weighting, reaching state of the art on LoRA and full fine-tuning merging benchmarks for CLIP ViTs and Llama-3 8B.

Problem

Layer-wise activation-based model merging computes all layer statistics from unmerged activations, so changes to early layers cause covariate shift that degrades the merged model when many task-specific fine-tuned models are combined without retraining.

Approach

Merge layers sequentially, recomputing each layer's input activations through the already-merged preceding layers (merging covariate shift correction), and weight tasks per layer by the off-diagonal correlation of normalized Gram matrices, using a closed-form regression solution.

Key results

• With LoRA on ViT-B/32, CoM reaches 92.40% normalized accuracy versus 76.43% for the second-best method CORE.
• With LoRA on Llama-3 8B, CoM reaches 99.50% normalized accuracy versus 94.16% for CORE (TSV-based).
• With LoRA on ViT-L/14, CoM reaches 91.06% normalized accuracy versus 86.21% for CORE; with full fine-tuning on ViT-B/32 it reaches 94.8% normalized, versus 93.2% for Iso-C.

Surprising findings

• RegMean underperforms simple task arithmetic on ViT-L/14 and Llama-3 8B despite being the closed-form activation-matching baseline CoM extends.
• Iso-C drops to 57.08% normalized accuracy on Llama-3 8B LoRA, far below other methods.

Limitations

• Relies on validation data samples to estimate activation Gram matrices, so it is data-dependent.
• Focuses only on activation matching for linear layers; other merging objectives are not explored.
• Gains are smaller with full fine-tuning, where closed-form activation matching is less effective.

Experimental setup

Datasets: Stanford Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN SNLI MultiNLI SICK SciTail

Benchmarks: 8-task CLIP vision merging benchmark (ViT-B/32, ViT-L/14) six-task NLI language merging benchmark with Llama-3 8B

Metrics: normalized accuracy absolute accuracy

Baselines: Task Arithmetic TIES DARE Consensus TA LiNeS FisherAVG RegMean MaTS TSV Iso-C KnOTS CORE

Builds on: RegMean Task Arithmetic ProDistill Iso-C KnOTS

Closest papers by meaning

• Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.90)
• $C^2M^3$: Cycle-Consistent Multi-Model Merging (NeurIPS 2024 · similarity 0.89)
• Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
• LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging (ICLR 2025 · similarity 0.88)
• Scalable Model Merging with Progressive Layer-wise Distillation (ICML 2025 · similarity 0.88)
• Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.87)