Research map / Architectures and efficiency / Pruning and efficient tuning
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Low-Rank Clone (LRC), a single-stage pre-training distillation method that learns low-rank projections to softly prune teacher weights into a student and aligns activations (including FFN) without extra alignment modules, matching SOTA SLMs with ~10-20B tokens.
Problem
Training high-performing small language models is computationally expensive; existing distillation and hard-pruning pipelines lose teacher information, need alignment modules, and underuse FFN activations.
Approach
Trainable low-rank projection matrices map teacher weights into student weights (soft pruning) and simultaneously align student activations with teacher activations via MSE, including FFN outputs; the same projections serve as implicit alignment layers, so no extra alignment matrices are needed.
Key results
- LRC-1.7B (20B tokens) reaches 64.98 average zero-shot vs 63.17 for Qwen3-1.7B trained on 36T tokens.
- LRC-1.5B (10B tokens) averages 63.48, above SmolLM2-1.7B trained on 11T tokens (60.50).
- LRC-4B (18B tokens) averages 70.30, on par with Qwen3-4B (70.29) and above Minitron-4B (68.31).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill with LRC rather than use Qwen3-1.7B | Qwen2.5-3B-Instruct to LRC-1.7B · Zero-shot downstream tasks (<2B; Avg.) | average score: 63.17 -> 64.98 | Helped |
| distill with LRC rather than use SmolLM2-1.7B | Llama-3.2-3B-Instruct to LRC-1.5B · Zero-shot downstream tasks (<2B; Avg.) | average score: 60.50 -> 63.48 | Helped |
| distill with LRC rather than use Qwen3-4B | Qwen2.5-7B-Instruct to LRC-4B · Zero-shot downstream tasks (>2B; Avg.) | average score: 70.29 -> 70.30 | No clear effect |
| use LRC instead of Sheared-Llama under the matched 2.7B setup | Llama-2-7B to 2.7B models · Zero-shot downstream tasks (>2B; Avg.) | average score: 52.55 -> 52.67 | No clear effect |
| remove FFN gate clone loss (add/retain FFN gate clone loss) (ablation) | LRC · LM loss convergence (2.5B tokens) | LM loss: 2.677 -> 2.639 | Helped |
| remove FFN down clone loss (add/retain FFN down clone loss) (ablation) | LRC · LM loss convergence (2.5B tokens) | LM loss: 2.651 -> 2.639 | Helped |
| remove FFN up clone loss (add/retain FFN up clone loss) (ablation) | LRC · LM loss convergence (2.5B tokens) | LM loss: 2.639 -> 2.639 | No clear effect |
| remove individual attention q, k, v, or o clone-loss term (ablation) | LRC · LM loss convergence (2.5B tokens) | LM loss: 2.639 -> 2.630; 2.639 -> 2.629; 2.639 -> 2.639; 2.639 -> 2.636 | Mixed |
| remove all attention clone losses (ablation) | LRC · LM loss convergence during training | LM loss | Mixed |
| remove all FFN clone losses (ablation) | LRC · LM loss convergence during training | LM loss | Hurt |
| remove all activation clone loss (ablation) | LRC · Training time to LM loss 3.0 | training time: more than 2× reduction in training time usage | Helped |
| add separate alignment matrices for attention and FFN outputs (ablation) | LRC · Final performance and training time | final performance | Hurt |
Surprising findings
- SFT on UltraChat lowered IFEval for LRC-4B (13.67 after SFT vs 36.09 before) and gave little safety gain.
- Sharing FFN projection matrices hurt performance more than sharing attention projections, suggesting FFNs need dedicated capacity.
Limitations
- Performance ceiling under larger-scale training is unexplored.
- Current implementation keeps the same FFN intermediate dimension for teacher and student.
- Few-shot performance gains are more modest than baselines; the paper reports no error bars due to compute cost.
Experimental setup
Datasets: Fineweb-Edu DCLM Cosmopedia V2 OpenHermes 2.5 UltraChat RedPajama
Benchmarks: ARC-E ARC-C LogiQA CSQA PIQA WinoGrande BoolQ SciQ MMLU ToxiGen TruthfulQA IFEval
Metrics: accuracy accuracy norm MC2 instance-level loose accuracy LM loss
Baselines: Sheared Llama Minitron-4B TinyBERT MiniCPM-1.2B SmolLM2-1.7B Qwen3-1.7B Qwen3-4B Gemma3-4B InternLM2-1.8B
Builds on: Minitron Sheared LLaMA TinyBERT LLM-Pruner SliceGPT
Closest papers by meaning
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.89)
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching (NeurIPS 2025 · similarity 0.89)
- LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation (ICLR 2025 · similarity 0.87)
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models (AAAI 2025 · similarity 0.87)
- On Teacher Hacking in Language Model Distillation (ICML 2025 · similarity 0.87)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.