AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Low-Rank Hierarchical Merging for Efficient Long-to-Short Reasoning

Zeqiu Yu, Xuesheng Zhang, Wenxiao Zhao, Shixiao Wang

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Low-Rank Hierarchical Merging (LHM) for long-to-short reasoning model merging, aiming to better account for loss-landscape response to parameter displacement than linear interpolation.

Problem

Long-to-short (L2S) model merging combines the reasoning strength of long chain-of-thought models with the concise style of base models, but existing linear-interpolation and first-order activation-statistics methods degrade accuracy on hard tasks.

Approach

A training-based merging method, Low-Rank Hierarchical Merging, that models how the loss landscape responds to parameter displacement (second-order-aware) using low-rank structure and hierarchical merging, rather than linear interpolation or first-order activation statistics.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add low-rank Hessian trace weighting (ablation)Task Arithmetic · 6 math benchmarks, 1.5Bavg accuracy (%): 41.5 -> 43.2Improved
add LSOS index (Hessian trace times task-vector norm) (ablation)Task Arithmetic + Hessian · 6 math benchmarks, 1.5Bavg accuracy (%): 43.2 -> 45.9Improved
add DCCM dual-constraint coefficient mapping (ablation)Task Arithmetic + LSOS · 6 math benchmarks, 1.5Bavg response length (tokens): 2631 -> 2239Improved
add DCCM (ablation)AIM + LSOS · 6 math benchmarks, 1.5Bavg accuracy (%): 46.8 -> 48.8Improved
use LHM instead of ORCA-AIMLHM-AIM · 6 math benchmarks, 1.5Bavg accuracy (%): 45.6 -> 48.8Improved
use LHM instead of ORCA-AIMLHM-AIM · 6 math benchmarks, 7Bavg accuracy (%): 75.1 -> 76.9Improved
lower global coefficient to 0.3 (ablation)LHM · 6 math benchmarks, 1.5Blength reduction / accuracy: up to 93.1% length reductionMixed
increase Hessian rank k beyond 16 (ablation)LHM · 6 math benchmarks, 1.5Bavg accuracyNo clear effect
vary bandwidth alpha from 0.10 to 0.20 (ablation)LHM · 6 math benchmarks, 1.5Bavg accuracy: within 0.3%No clear effect
use larger bandwidth alpha (ablation)LHM · 6 math benchmarks, 1.5Bavg accuracyHurt

Experimental setup

Metrics: accuracy

Baselines: linear interpolation activation-statistics-based L2S merging methods

Builds on: linear interpolation merging activation-statistics merging

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy