Research map / Learning theory and algorithms / Matrix and tensor methods

Generalizing the Geometry of Model Merging Through Fréchet Averages

NeurIPS 2026 · Learning theory and algorithms › Matrix and tensor methods · approach: Randomized sketching for matrices

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Casts model merging as a symmetry-invariant Fréchet mean on a quotient manifold, shows Fisher merging arises as a quadratic surrogate of this objective, and applies a quotient-aware geodesic merge with a rank-lift to LoRA adapters on ViT-B/32 vision tasks and Llama 3 8B NLI tasks.

Problem

Weight-space model merging, especially for LoRA adapters, is fragile because parameter-space averaging ignores architectural and low-rank gauge symmetries, so equivalent parameterizations of the same model yield different or off-orbit merges.

Approach

Formulate merging as a Fréchet mean on the quotient of parameter space by the symmetry group (GL(r) or O(r) for LoRA), using orbit alignment (gauge choice) followed by horizontal geodesic averaging; show Fisher merging is a quadratic surrogate of the Fisher-Rao Fréchet objective; and add a quotient-compatible rank lift for merging LoRA adapters.

Key results

• On ViT-B/32 eight-task LoRA merging, GeoMerge+lift reaches 77.10% average normalized accuracy, above TSV+Iso-C on Core lift (76.43%) and the best reported baselines.
• On Llama 3 8B NLI tasks, GeoMerge with Task Arithmetic achieves 91.88% average normalized accuracy versus 90.38% for standard TA.
• The Euclidean TSV+Iso-C method applied on the same lift reaches 75.97%, underperforming the Core lift baseline, so the gains are attributed to the geometric averaging rather than the lift alone.

Surprising findings

• Euclidean averaging on the same lifted adapters underperforms the Core Space lift, suggesting the lift alone does not explain gains.
• Naive averaging of two gauge-equivalent checkpoints (theta and -theta) collapses to zero, and Fisher-weighted averaging shows the same failure in the toy model.

Limitations

• The high-rank lift is a simple heuristic with no optimality claim, and better lift choices are left to future work.
• Integration with magnitude/sign-based methods like TIES and DARE-TIES is not done since they are coordinate-dependent.
• The NLI experiment only runs the TA variant, not the full stack, due to limited GPU resources; computational efficiency could still be improved.

Experimental setup

Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC SUN397 SVHN SNLI MNLI SICK QNLI

Benchmarks: Cars DTD EuroSAT GTSRB MNIST RESISC SUN397 SVHN SNLI MNLI SICK QNLI

Metrics: normalized accuracy

Baselines: Task Arithmetic TIES KnOTS Core Space Iso-C TSV+Iso-C

Builds on: Fisher merging KnOTS Core Space Task Arithmetic TSV Iso-C

Closest papers by meaning

• Model merging with SVD to tie the Knots (ICLR 2025 · similarity 0.86)
• CAMEx: Curvature-aware Merging of Experts (ICLR 2025 · similarity 0.84)
• Accurate and Efficient Low-Rank Model Merging in Core Space (NeurIPS 2025 · similarity 0.83)
• $C^2M^3$: Cycle-Consistent Multi-Model Merging (NeurIPS 2024 · similarity 0.83)
• PLeaS - Merging Models with Permutations and Least Squares (CVPR 2025 · similarity 0.83)
• StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold (NeurIPS 2025 · similarity 0.83)