Research map / Large language models / Retrieval and knowledge
Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes parametric knowledge transfer: sensitivity-based extraction of task-specific submatrices from a larger LLaMA teacher, injected into a smaller student's LoRA module, showing gains over plain LoRA fine-tuning on four benchmarks.
Problem
Whether task-specific implicit knowledge stored in the parameters of a larger LLM can be transferred to smaller LLMs of different scale, as an alternative to output-based distillation.
Approach
Extract task-relevant parameters from a teacher via sensitivity scores (first-order Taylor approximation on ~32 seed samples), select layers and submatrices matching the student's dimensions, factorize them with SVD, and inject them into the student's LoRA initialization (with the extracted part subtracted from the base weights so training starts at the pretrained point).
Key results
- LLaMA-1 30B to 7B transfer raises the 7B-LoRA average from 27.00 to 28.63 (+6.04%).
- LLaMA-2 13B to 7B transfer raises the average from 32.28 to 34.47 (+6.78%).
- Sensitivity-based submatrix extraction beats random extraction and single-weight transfer; single-weight transfer performs near no-transfer baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer sensitivity-extracted parameters from LLaMA-1 30B instead of using 7B-LoRA alone | LLaMA-1 7B · four-benchmark average | average score: 27.00 -> 28.63 | Helped |
| transfer sensitivity-extracted parameters from LLaMA-2 13B instead of using 7B-LoRA alone | LLaMA-2 7B · four-benchmark average | average score: 32.28 -> 34.47 | Helped |
| increase teacher scale from 13B to 30B | LLaMA-1 7B · GSM | accuracy: 18.73 -> 18.63 | Hurt |
| extract parameters using task-specific seed data rather than generic transfer data (ablation) | LLaMA-2 7B · GSM 0-shot | accuracy: 23.38 -> 25.30 | Helped |
| initialize LoRA with the proposed weight-preserving strategy rather than the LoRA-paper initialization (ablation) | LLaMA-1 7B · four-benchmark average | average score | Helped |
| increase the number of sensitivity seed samples (ablation) | student model · knowledge transfer | performance variance | Helped |
| select teacher layers by sensitivity rather than random, top, or last-layer selection (ablation) | LLaMA-1 teacher-to-student transfers · knowledge transfer | student performance | Helped |
| transfer all available modules rather than the embedding layer alone (ablation) | student model · knowledge transfer | student performance | Helped |
| transfer coherent submatrices rather than isolated single weights (ablation) | student model · knowledge transfer | student performance | Helped |
| use sensitivity-based rather than random submatrix extraction (ablation) | LLaMA-1 13B to 7B · four-benchmark average | average score: 27.83 -> 28.22 | Helped |
| use sensitivity-based rather than random submatrix extraction (ablation) | LLaMA-1 30B to 7B · four-benchmark average | average score: 28.11 -> 28.63 | Helped |
| increase LoRA rank beyond 16 when transferring all modules (ablation) | LLaMA-1 7B · four-benchmark average | average score: 28.22 -> 28.27; 28.63 -> 28.54 | No clear effect |
Surprising findings
- Transferring isolated single weights, whether random or sensitivity-selected, yields results comparable to no transfer at all.
- Transferring from LLaMA-2 70B to 7B (restricted to FFN and embedding) gives larger gains than from 13B to 7B.
Limitations
- Gains are modest in absolute terms (e.g., GSM and MMLU differences are about 1-2 points).
- Transfer requires matching or compatible architectures; attention structure differences (GQA in 70B) required restricting the transfer to FFN and embedding modules.
- Only LLaMA family models and four benchmarks were studied.
Experimental setup
Datasets: GSM8K MMLU Super-NaturalInstructions LIMA AlpacaFarm
Benchmarks: GSM8K MMLU Super-NaturalInstructions AlpacaFarm
Metrics: accuracy exact match ROUGE-L win rate
Baselines: LoRA fine-tuning (no transfer) KD SeqKD Gaussian LoRA initialization random submatrix extraction
Builds on: LoRA knowledge distillation first-order Taylor parameter sensitivity pruning
Closest papers by meaning
- Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.89)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.89)
- KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models (ICLR 2025 · similarity 0.89)
- The Rise of Parameter Specialization for Knowledge Storage in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.88)
- LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.