Research map / Large language models / Reasoning and chain of thought
Outlier Matters: Efficient Long-to-Short Reasoning via Outlier-Guided Model Merging
Qiyuan Zhu, Dezhi Li, Lujun Li, Xiaoyu Qin, Wei Li, Hao Gu, Hua Xu, Sirui Han, Yike Guo
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ORCA, an outlier-guided model merging framework that fuses base and reasoning LLMs to produce shorter reasoning chains (long-to-short reasoning).
Problem
Large reasoning language models produce extensive reasoning chains that incur high computational overhead; the goal is to shorten reasoning while preserving accuracy.
Approach
A plug-and-play model merging method that uses outlier activation patterns to guide how base models and reasoning models are fused, yielding concise reasoning without retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add outlier-guided layer-wise allocation (ablation) | Task Arithmetic (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): 53.6 -> 54.9 | Improved |
| outlier-guided instead of random allocation (ablation) | Task Arithmetic (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): 52.7 -> 54.9 | Improved |
| increasing layer-wise allocation instead of uniform (ablation) | Task Arithmetic (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): 53.6 -> 45.6 | Hurt |
| decreasing layer-wise allocation instead of uniform (ablation) | Task Arithmetic (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): 53.6 -> 47.2 | Hurt |
| add magnitude-based ORCA probe (ablation) | Task Arithmetic (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): 53.6 -> 54.2 | Improved |
| random probe selection (ablation) | Task Arithmetic (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): 53.6 -> 53.9 | No clear effect |
| probe low-magnitude (minimum value) activations (ablation) | Task Arithmetic (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): 53.6 -> 53.1 | Hurt |
| add adaptive alignment factor (ablation) | four merging baselines (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): +1.2% | Improved |
| add outlier-guided coefficient (ablation) | four merging baselines (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): +1.5% | Improved |
| add probe signals (ablation) | four merging baselines (Qwen2.5-7B) · Six math benchmarks average | accuracy (%): +0.6% | Improved |
| add ORCA to Task Arithmetic | Qwen2.5-7B merge · Six math benchmarks average | accuracy (%): 53.6 -> 55.7 | Improved |
| add ORCA to TIES-Merging | Qwen2.5-7B merge · Six math benchmarks average | accuracy (%): 54.5 -> 56.3 | Improved |
Closest papers by meaning
- Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilities (AAAI 2026 · similarity 0.89)
- Mitigating Overthinking in Large Reasoning Language Models via Reasoning Path Deviation Monitoring (NeurIPS 2026 · similarity 0.88)
- Hint Tuning: Less Data Makes Better Reasoners (NeurIPS 2026 · similarity 0.88)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.88)
- FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning (ICLR 2026 · similarity 0.88)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
