Research map / Architectures and efficiency / Attention and transformers
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces rotation symmetry for transformer self-attention and a closed-form rotation-based parameter matching module that plugs into model fusion methods (Simple, Fisher, RegMean, OT) to improve merged model performance.
Problem
Permutation symmetry is discrete and only fully covers MLP/FFN blocks, so parameter matching for transformer model fusion is limited; the self-attention layers lack an effective continuous symmetry-based alignment.
Approach
Extends permutation symmetry to continuous rotation symmetry for Q/K and V/O matrices in self-attention, and derives a closed-form (SVD/Procrustes-based) optimal rotation for matching two attention layers, combined with rescaling symmetry and Hungarian-based FFN permutation matching as a plug-and-play step before fusion.
Key results
- On in-domain NER with DeBERTa, Simple merging improves from 27.54 to 31.31 F1 with matching; on NER-CoNLL OOD DeBERTa Simple rises from 10.80 to 23.30.
- On ViT CIFAR-10 merging, OT-ACTS achieves 61.15 without matching and 61.25 with matching, showing modest gains for strong baselines.
- Matching overhead is under 5% of Fisher/RegMean merging runtime (e.g., 1.59 s matching vs 197 s Fisher for DeBERTa).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add parameter matching | Fisher fusion · Emotion with RoBERTa | macro F1: 44.02 -> 44.05 | Helped |
| add parameter matching | Regmean fusion · Emotion with RoBERTa | macro F1: 35.87 -> 39.95 | Helped |
| add parameter matching | Simple fusion · NER with DeBERTa | micro F1: 27.54 -> 31.31 | Helped |
| add parameter matching | Fisher fusion · NER with DeBERTa | micro F1: 33.20 -> 33.83 | Helped |
| add parameter matching | Simple fusion · NER-CoNLL with DeBERTa | micro F1: 10.80 -> 23.30 | Helped |
| add parameter matching | Regmean fusion · NER-CoNLL with RoBERTa | micro F1: 26.86 -> 26.87 | Helped |
| add rotation-based matching and rescaling | ViT fusion baselines · CIFAR-10 | accuracy: 7.60 -> 10.19; 17.96 -> 18.58; 14.24 -> 15.35; 32.08 -> 32.53; 61.15 -> 61.25; 57.11 -> 57.17 | Helped |
| add rescaling after matching (ablation) | ViT fusion baselines · CIFAR-10 | accuracy: 10.22 -> 10.19; 18.61 -> 18.58; 15.31 -> 15.35; 32.50 -> 32.53; 61.23 -> 61.25; 57.16 -> 57.17 | Mixed |
| match only attention modules, excluding FFN modules (ablation) | ViT model matching · CIFAR-10 | accuracy | Mixed |
| match only FFN modules, excluding attention modules (ablation) | ViT model matching · CIFAR-10 | accuracy | Mixed |
| match the first 5 attention layers rather than tail layers (ablation) | DeBERTa matching · GLUE CoLA and STS-B | evaluation loss | Helped |
Surprising findings
- Gains are small or absent for some strong baselines (e.g., OT-ACTS improves only 0.1 points on ViT), and the Roberta Emotion results show no change for Simple merging.
- FFN-only matching (w/o ATTN) is reported as less effective for transformers than for MLPs/CNNs, suggesting attention matching drives most of the benefit.
Limitations
- Multi-model matching lacks global optimality guarantees; pairwise anchor-based or iterative matching is path dependent.
- Rotation and rescaling are optimized sequentially, not jointly, so global optimality over both is not guaranteed.
- Only rotation, permutation, and rescaling symmetries are considered; other transformer symmetries (normalization, softmax) are not characterized.
Experimental setup
Datasets: Emotion classification datasets (emoint, ssec, electoraltweets, grounded_emotions, affectivetext, and others) OntoNotes CoNLL GLUE (STS-B, SST-2, CoLA) CIFAR-10
Benchmarks: Emotion classification NER (OntoNotes) NER-CoNLL out-of-domain CIFAR-10 ViT merging GLUE CoLA and STS-B (subset layer matching)
Metrics: macro F1 micro F1 accuracy Euclidean distance loss barrier evaluation loss runtime
Baselines: Simple averaging Fisher merging RegMean OT-ACTS OT-ACTS-EMD OT-WTS OTFusion Git Re-Basin
Builds on: Git Re-Basin OTFusion RegMean Fisher merging Kabsch algorithm
Closest papers by meaning
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.89)
- Permutation Equivariance of Transformers and Its Applications (CVPR 2024 · similarity 0.88)
- Symmetry Induces Structure and Constraint of Learning (ICML 2024 · similarity 0.86)
- On Making $SE(2)$-Invariant Networks Optimal (NeurIPS 2026 · similarity 0.86)
- Learning Infinitesimal Generators of Continuous Symmetries from Data (NeurIPS 2024 · similarity 0.85)
- When GNNs meet symmetry in ILPs: an orbit-based feature augmentation approach (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.