Research map / Large language models / Reasoning and chain of thought
Low-Rank Hierarchical Merging for Efficient Long-to-Short Reasoning
Zeqiu Yu, Xuesheng Zhang, Wenxiao Zhao, Shixiao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Low-Rank Hierarchical Merging (LHM) for long-to-short reasoning model merging, aiming to better account for loss-landscape response to parameter displacement than linear interpolation.
Problem
Long-to-short (L2S) model merging combines the reasoning strength of long chain-of-thought models with the concise style of base models, but existing linear-interpolation and first-order activation-statistics methods degrade accuracy on hard tasks.
Approach
A training-based merging method, Low-Rank Hierarchical Merging, that models how the loss landscape responds to parameter displacement (second-order-aware) using low-rank structure and hierarchical merging, rather than linear interpolation or first-order activation statistics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add low-rank Hessian trace weighting (ablation) | Task Arithmetic · 6 math benchmarks, 1.5B | avg accuracy (%): 41.5 -> 43.2 | Improved |
| add LSOS index (Hessian trace times task-vector norm) (ablation) | Task Arithmetic + Hessian · 6 math benchmarks, 1.5B | avg accuracy (%): 43.2 -> 45.9 | Improved |
| add DCCM dual-constraint coefficient mapping (ablation) | Task Arithmetic + LSOS · 6 math benchmarks, 1.5B | avg response length (tokens): 2631 -> 2239 | Improved |
| add DCCM (ablation) | AIM + LSOS · 6 math benchmarks, 1.5B | avg accuracy (%): 46.8 -> 48.8 | Improved |
| use LHM instead of ORCA-AIM | LHM-AIM · 6 math benchmarks, 1.5B | avg accuracy (%): 45.6 -> 48.8 | Improved |
| use LHM instead of ORCA-AIM | LHM-AIM · 6 math benchmarks, 7B | avg accuracy (%): 75.1 -> 76.9 | Improved |
| lower global coefficient to 0.3 (ablation) | LHM · 6 math benchmarks, 1.5B | length reduction / accuracy: up to 93.1% length reduction | Mixed |
| increase Hessian rank k beyond 16 (ablation) | LHM · 6 math benchmarks, 1.5B | avg accuracy | No clear effect |
| vary bandwidth alpha from 0.10 to 0.20 (ablation) | LHM · 6 math benchmarks, 1.5B | avg accuracy: within 0.3% | No clear effect |
| use larger bandwidth alpha (ablation) | LHM · 6 math benchmarks, 1.5B | avg accuracy | Hurt |
Experimental setup
Metrics: accuracy
Baselines: linear interpolation activation-statistics-based L2S merging methods
Builds on: linear interpolation merging activation-statistics merging
Closest papers by meaning
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.89)
- TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning (NeurIPS 2026 · similarity 0.88)
- Activation-Guided Consensus Merging for Large Language Models (NeurIPS 2025 · similarity 0.88)
- RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Prior (AAAI 2026 · similarity 0.88)
- Mixture-Trained Merging for Unified Multi-Objective Models (NeurIPS 2026 · similarity 0.88)
- Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
