AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization

Junlin He, Yihong Tang, Tong Nie, Guilong Li, Binyu Yang, Jinxiao Du, Lijun Sun, Wei Ma

ICML 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add activation-aware channel-selection initialization of projection matricesLRC-1.5B (Llama3.2-3B teacher) · GSM8Kaccuracy: 0.21 -> 0.44Improved
add activation-aware initializationLRC-1.7B (Qwen2.5-3B teacher) · GSM8Kaccuracy: 0.09 -> 0.42Improved
add activation-aware initializationLRC-2.7B (Llama2-7B teacher) · Reasoning averageaccuracy: 0.06 -> 0.10Improved
add activation-aware initializationLRC-4B (Qwen2.5-7B teacher) · Reasoning averageaccuracy: 0.18 -> 0.33Improved
add activation-aware initializationLRC-4B · General ability averageaccuracy: 0.69 -> 0.69No clear effect
add activation-aware initializationLRC-1.5B (Llama3.2-3B teacher) · General ability averageaccuracy: 0.59 -> 0.61No clear effect
add activation-aware initialization (ablation)linear autoencoder proxy · reconstruction lossloss: 1.24 -> 0.08Improved
add activation-aware initialization (ablation)linear autoencoder proxy · hidden eRankeRank: 81.30 -> 263.28Improved
use random orthogonal init instead of activation-aware init (ablation)LRC distillation · GSM8Kaccuracy: 0.44 -> 0.30Hurt
add orthogonal regularization (1e-4) to random init (ablation)LRC distillation · GSM8Kaccuracy: 0.21 -> 0.26Improved
raise distillation temperature from 1 to 40 (ablation)RED-1.5B · GSM8Kaccuracy: 0.44 -> 0.08Hurt
raise distillation temperature from 1 to 40 (ablation)RED-1.5B · MMLUaccuracy: 0.50 -> 0.52Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy