Research map / Continual learning / Continual fine-tuning of LLMs
Orthogonal Model Merging
Sihan Yang, Kexuan Shi, Weiyang Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add magnitude correction to Lie-algebra averaging (ablation) | OrthoMerge · OFT experts, Llama-3.1-8B in-domain | average accuracy: 41.01 -> 46.25 | Improved |
| average in Lie algebra instead of averaging orthogonal matrices directly (ablation) | OrthoMerge · OFT experts, Llama-3.1-8B in-domain | average accuracy: 39.53 -> 41.01 | Improved |
| merge rotations by sequential product instead of Lie-algebra averaging (ablation) | OrthoMerge · OFT experts, Llama-3.1-8B MATH500 | MATH500: 24.60 -> 8.60 | Hurt |
| merge OFT experts with OrthoMerge instead of TIES in weight space | OrthoMerge · OFT experts, Llama-3.1-8B in-domain | average accuracy: 44.61 -> 46.25 | Improved |
| merge with OrthoMerge instead of TSV-M on OFT experts | OrthoMerge · OFT experts, M-ARC and AGIEval avg | average accuracy: 40.48 -> 41.80 | Improved |
| add OrthoMerge-G decoupling to Task Arithmetic | TA · MergeBench Llama-3.2-3B task avg | average score: 34.16 -> 35.48 (+1.32) | Improved |
| add OrthoMerge-C decoupling to TIES | TIES · MergeBench Llama-3.2-3B task avg | average score: 39.16 -> 41.57 (+2.41) | Improved |
| add OrthoMerge-G decoupling to TSV-M | TSV-M · MergeBench Llama-3.2-3B task avg | average score: 41.52 -> 40.03 (-1.49) | Hurt |
| add OrthoMerge-C decoupling to TSV-M | TSV-M · MergeBench Llama-3.2-3B task avg | average score: 41.52 -> 42.07 (+0.55) | Improved |
| add OrthoMerge-C decoupling to TIES | TIES · MergeBench MMLU | MMLU (excl. math/coding): 54.52 -> 55.36 | Improved |
| add OrthoMerge-C decoupling to TIES | TIES · Qwen2.5-VL-7B task avg | average accuracy: 63.54 -> 64.04 (+0.50) | Improved |
| add OrthoMerge-G decoupling to TSV-M | TSV-M · Qwen2.5-VL-7B IFEval | IFEval: 54.90 -> 57.86 | Improved |
Closest papers by meaning
- Group and Shuffle: Efficient Structured Orthogonal Parametrization (NeurIPS 2024 · similarity 0.86)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.86)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.85)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.85)
- Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging (NeurIPS 2026 · similarity 0.85)
- CAMEx: Curvature-aware Merging of Experts (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
