Research map / Architectures and efficiency / RNNs and normalization
$C^2M^3$: Cycle-Consistent Multi-Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C²M³, a data-free multi-model weight-matching method that uses Frank-Wolfe over all layers jointly and object-to-universe permutation factorization to guarantee cycle consistency, then averages in the universe space with REPAIR renormalization.
Problem
Merging n≥3 neural networks in weight space requires neuron permutations that are cycle-consistent; existing pairwise, layer-wise matching methods accumulate error along permutation cycles and depend on layer order and mapping direction.
Approach
Factorize each pairwise permutation as P^{pq} = P^p (P^q)^T via a shared universe space, so cycle consistency holds by construction; optimize all layers jointly with a Frank-Wolfe solver on a higher-order objective, then merge by averaging in the universe space.
Key results
- On merging 5 models, C²M³ with REPAIR reaches 0.60 test accuracy on ResNet20 2x EMNIST... (table reports 0.60 vs 0.03 for MergeMany on ResNet 2x EMNIST) and 0.64 on VGG16 CIFAR10 vs 0.10 for MergeMany without REPAIR.
- Accuracy gains of up to about +20% over MergeMany across settings when merging n models, with cycle-consistent permutations that exactly recover the source model under cyclic mapping, unlike Git Re-Basin.
- In pairwise matching, Frank-Wolfe matches Git Re-Basin's loss barrier (e.g., 0.73 vs 0.73 train on ResNet 2x CIFAR10) with zero variance across seeds, while Git Re-Basin varies by up to 10% in accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply REPAIR after C²M³ merging (ablation) | C²M³ · CIFAR10 test (ResNet 2×) | accuracy: 0.40 -> 0.69 | Helped |
| Apply REPAIR after C²M³ merging (ablation) | C²M³ · CIFAR100 test (ResNet 4×) | accuracy: 0.30 -> 0.46 | Helped |
| Apply REPAIR after C²M³ merging (ablation) | C²M³ · CIFAR10 test (VGG16) | accuracy: 0.11 -> 0.62 | Helped |
| Apply REPAIR after C²M³ merging (ablation) | C²M³ · EMNIST test (MLP) | accuracy: 0.87 -> 0.87 | No clear effect |
| Apply REPAIR after C²M³ merging (ablation) | C²M³ · CIFAR100 test (ResNet 16×) | accuracy: 0.39 -> 0.49 | Helped |
| Use Frank-Wolfe global matching instead of Git Re-Basin | Pairwise ResNet 8× matching · CIFAR100 | test loss barrier: 1.54 ± 0.13 -> 1.42 ± 0.10 | Helped |
| Use Frank-Wolfe global matching instead of Git Re-Basin | Pairwise ResNet20 2× with BatchNorm · CIFAR10 | test loss barrier: 4.62 ± 0.65 -> 3.79 ± 0.57 | Helped |
| Initialize permutations to the barycenter rather than identity (ablation) | Frank-Wolfe pairwise matching · ResNet20 2×, models (a, b) | test loss barrier: $0.52$ -> $\mathbf{0.47}$ | Helped |
| Use Sinkhorn initialization rather than identity (ablation) | Frank-Wolfe pairwise matching · ResNet20 2×, models (a, b) | test loss barrier: $0.52$ -> $0.60 \pm 0.04$ | Hurt |
Surprising findings
- Fixing one source model as the universe reference yields severe oscillations, with one model reaching as low as 65% accuracy.
- Merged accuracy does not decrease monotonically with number of models for ResNet20 4x; composition of the model set seems to matter more than cardinality.
Limitations
- Merging is data-free and relies on weight similarity; performance depends on width and architecture, with narrow models merging poorly.
- Runtime grows substantially for wide models (about 716s for 16x ResNet20 with n=3) compared with MergeMany on small settings.
- Experiments are limited to small image classifiers (MLP, ResNet20, VGG16) on MNIST-family and CIFAR datasets.
Experimental setup
Datasets: MNIST EMNIST CIFAR10 CIFAR100
Benchmarks: MNIST EMNIST CIFAR10 CIFAR100
Metrics: accuracy loss loss barrier cosine similarity wall-clock time
Baselines: Git Re-Basin MergeMany Naive averaging REPAIR
Builds on: Git Re-Basin REPAIR MergeMany Frank-Wolfe algorithm
Closest papers by meaning
- Rethinking Layer-wise Model Merging through Chain of Merges (NeurIPS 2026 · similarity 0.89)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.87)
- Accurate and Efficient Low-Rank Model Merging in Core Space (NeurIPS 2025 · similarity 0.86)
- PLeaS - Merging Models with Permutations and Least Squares (CVPR 2025 · similarity 0.85)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.85)
- Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.