Research map / Architectures and efficiency / Pruning and efficient tuning
CAMEx: Curvature-aware Merging of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAMEx, a curvature-aware expert merging method for sparse mixture-of-experts that uses a learned Kronecker-factored curvature matrix (natural-gradient inspired) per expert, plus a dynamic merging architecture with a global expert passed across layers.
Problem
Expert merging in sparse mixture-of-experts (SMoE) models assumes a flat Euclidean parameter space, which may limit generalization during pre-training and fine-tuning; existing curvature-aware (Fisher-based) merging is memory and compute heavy.
Approach
Insert a learnable curvature matrix per expert, approximated as a low-rank sum of Kronecker products, into the merging rule (CA-Merg) so merging follows a natural-gradient-like update, learned via backpropagation without computing or inverting the Fisher matrix; a dynamic architecture passes a global merged expert across layers to cut parameters.
Key results
- On WikiText-103 pre-training, Domain-specific-CA reaches perplexity 21.50 vs 23.03 for Vanilla and 22.42 for SMoE, and the Dynamic architecture reaches 21.55 with 470M params vs 522M.
- On GLUE fine-tuning with T5-base, Ties-CA achieves the best scores on SST-2 (94.61), MRPC (92.49), CoLA (60.06) and MNLI (86.45).
- On ImageNet-1k with Swin-T small, Ties-CA and Dare-CA reach 83.38 top-1 accuracy vs 83.14 for Vanilla.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add curvature-aware merging to Ties | T5-base on GLUE · GLUE SST-2 | accuracy: 93.92 -> 94.61 | Helped |
| Add curvature-aware merging to Ties | T5-base on GLUE · GLUE MRPC | accuracy: 91.44 -> 92.49 | Helped |
| Add curvature-aware merging to Ties | T5-base on GLUE · GLUE MNLI | accuracy: 86.39 -> 86.45 | No clear effect |
| Add curvature-aware merging to domain-specific merging | GPT-2 small · WikiText-103 pre-training | perplexity: 21.64 -> 21.50 | Helped |
| Use dynamic cross-layer merging | GPT-2 small · WikiText-103 pre-training | perplexity: 21.64 -> 21.55 | Helped |
| Add curvature-aware merging to domain-specific merging | GPT-2 small · WikiText-2 supervised fine-tuning | perplexity: 21.56 -> 21.06 | Helped |
| Add curvature-aware merging to DARE | T5-base · SQuAD | EM: 80.88 -> 81.76 | Helped |
| Add curvature-aware merging to Ties | Swin-Transformer small · ImageNet-1k | Acc@1: 83.28 -> 83.38 | Helped |
| Increase Kronecker rank (ablation) | Domain-specific-CA, Ties-CA, and Dare-CA · STSB, MRPC, and RTE | task performance | Mixed |
| Use scaling factor α in [0.8, 1] (ablation) | Curvature-aware variants · STSB, MRPC, and RTE | task performance | Helped |
Surprising findings
- Performance of Ties-CA and Dare-CA declines as the Kronecker rank increases, which the authors attribute to interference between the masking mechanism and curvature-matrix learning.
- Ties-CA gains on MNLI over the non-CA Ties variant are not statistically significant (p=0.16) per the authors' t-test.
Limitations
- Curvature-aware merging shows reduced compatibility with Ties and Dare at higher Kronecker ranks.
- Kronecker approximation of the curvature matrix may limit its representational capacity, as acknowledged by the authors.
- Gains on several GLUE tasks are small (e.g., MNLI differences of about 0.1 points) and many results are within run-to-run variance.
Experimental setup
Datasets: WikiText-103 WikiText-2 GLUE SQuAD WikiQA ImageNet-1k
Benchmarks: WikiText-103 WikiText-2 GLUE SQuAD WikiQA ImageNet-1k ImageNet-A ImageNet-O ImageNet-R
Metrics: perplexity accuracy F1 exact match top-1 accuracy
Baselines: Vanilla SMoE Domain-specific merging TIES-merging DARE Twin-Merging
Builds on: SMEAR Lory TIES-merging DARE Fisher merging
Closest papers by meaning
- Curvature Tuning: Provable Training-free Model Steering From a Single Parameter (NeurIPS 2025 · similarity 0.86)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.86)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.86)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.85)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.85)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.