Research map / Architectures and efficiency / RNNs and normalization
Generalized Linear Mode Connectivity for Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes linear mode connectivity alignment beyond permutations (semi-permutation, orthogonal, invertible maps) and shows zero/low-barrier interpolation between independently trained ViTs and small GPT-2 models, including multi-model and width-heterogeneous settings.
Problem
Permutation-only re-basin fails to align independently trained Transformers, leaving barriers on interpolation paths because Transformers have richer symmetries (orthogonal residual stream, invertible QK/OV circuits, semi-permutation heads).
Approach
A unified symmetry framework (permutation, semi-permutation, orthogonal, invertible) for function-preserving reparameterizations of Transformers, combined with weight matching (Procrustes for residual stream, SOBLAP/LAP for FF and heads) and learned matching via straight-through estimators, extended to multi-model universe matching and width-heterogeneous alignment.
Key results
- Learned matching with the proposed symmetries reaches 0.00 barrier on CIFAR-10, CIFAR-100 and Tiny ImageNet ViTs, and 0.02 on Tiny Shakespeare GPT-2 (vs 2.84 vanilla averaging on Tiny ImageNet).
- Weight matching (training-free, data-free) reduces barriers substantially, e.g., 0.36 on CIFAR-10 ViT vs 1.69 for vanilla averaging, and 1.56 vs 4.34 on BookCorpus.
- On BookCorpus GPT-2, learned matching achieves 0.42 barrier, the only dataset where a non-zero barrier remains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned matching instead of weight matching | ViT alignment · CIFAR-10 | loss barrier: 0.36 ± 0.01 -> 0.00 ± 0.00 | Helped |
| Use full symmetry-aware learned matching instead of permutation-only matching (ablation) | ViT alignment · CIFAR-100 | loss barrier: 0.53 ± 0.07 -> 0.00 ± 0.00 | Helped |
| Use full symmetry-aware learned matching instead of permutation-only matching (ablation) | ViT alignment · Tiny ImageNet | loss barrier: 0.29 ± 0.02 -> 0.00 ± 0.00 | Helped |
| Use learned matching instead of weight matching | GPT-2 alignment · Tiny Shakespeare | loss barrier: 0.34 ± 0.01 -> 0.02 ± 0.00 | Helped |
| Use learned matching instead of weight matching | GPT-2 alignment · BookCorpus | loss barrier: 1.56 ± 0.02 -> 0.42 ± 0.01 | Helped |
| Use full symmetry-aware learned matching instead of permutation-only matching (ablation) | GPT-2 alignment · Tiny Shakespeare | loss barrier: 0.63 ± 0.17 -> 0.02 ± 0.00 | Helped |
| Use full symmetry-aware learned matching instead of permutation-only matching (ablation) | GPT-2 alignment · BookCorpus | loss barrier: 1.60 ± 0.04 -> 0.42 ± 0.01 | Helped |
| Initialize learned matching with weight matching rather than identity initialization (ablation) | ViT learned matching · CIFAR-10 | training and test loss | Helped |
| Add orthogonal matching rather than use permutation-based matching alone (ablation) | Data-free ViT weight matching · CIFAR-10 | loss barrier | Helped |
| Increase the number of weight-matching iterations (ablation) | Permutation-based ViT weight matching · CIFAR-10 | loss barrier | No clear effect |
| Sample interpolation coefficients narrowly and uniformly from [0.4, 0.6] instead of broadly or Gaussian sampling (ablation) | ViT learned matching · CIFAR-10 | interpolation loss barrier | Helped |
| Use fixed interpolation at λ = 0.5 instead of narrow uniform sampling (ablation) | ViT learned matching · CIFAR-10 | interpolation loss barrier | Hurt |
Surprising findings
- Learned matching with permutations only performs comparably to weight matching, while orthogonal residual alignment is what closes most of the gap.
- Language model barriers on BookCorpus remain non-zero, which the authors attribute possibly to genuinely distinct solutions rather than imperfect alignment.
Limitations
- Language experiments use small GPT-2 variants rather than larger contemporary LMs.
- Soft permutations can break exact functional equivalence at the aligned endpoint.
- Scope limited to pairs (and multi-model) of models pretrained on the same task.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet Tiny Shakespeare BookCorpus
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet Tiny Shakespeare BookCorpus
Metrics: loss barrier test loss along interpolation path
Baselines: Vanilla averaging Activation matching Weight matching Learned matching (permutations)
Builds on: Git Re-Basin OT fusion Sinkhorn Re-Basin Transformer circuits (QK/OV)
Closest papers by meaning
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.91)
- Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion (ICML 2025 · similarity 0.89)
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.89)
- Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries (ICML 2025 · similarity 0.89)
- Input Space Mode Connectivity in Deep Neural Networks (ICLR 2025 · similarity 0.88)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.