Research map / Large language models / Instruction and fine-tuning
LoKI: Low-Damage Knowledge Implanting of Large Language Models
Runyu Wang, Peng Ping, Zhengyu Guo, Xiaoye Zhang, Quan Shi, Liting Zhou, Tianbo Ji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use layer-balanced selection instead of global highest-attribution selection (ablation) | LoKI (q=10) · LB Reranker, 6 general benchmarks | avg degradation (%): 39.04% -> 8.86% | Improved |
| use layer-balanced selection instead of global lowest-attribution selection (ablation) | LoKI (q=10) · LB Reranker, 6 general benchmarks | avg degradation (%): 30.48% -> 8.86% | Improved |
| suppress highest-attribution KVA vectors vs lowest-attribution vectors (validates KVA) (ablation) | Llama3.1-8B-Instruct · 6 general benchmarks | avg degradation (%): 36.73% (S-H) vs 11.35% (S-L) | Improved |
| fine-tune with LoKI (q=30) instead of DoRA | Llama3.1-8B-Instruct on ToolACE · 6 general benchmarks | avg degradation (%): 4.92% -> 1.23% | Improved |
| increase q from 10 to 30 (ablation) | LoKI on ToolACE · BFCL V3 overall | overall accuracy (%): 56.76 -> 58.93 | Improved |
| increase q from 10 to 30 (ablation) | LoKI on ToolACE · 6 general benchmarks | avg degradation (%): 0.34% -> 1.23% | Hurt |
| increase q from 5 to 30 (ablation) | LoKI on LB Reranker · BEIR MAP@1 diff vs full fine-tune | relative difference (%): -3.1 -> +1.0 | Improved |
| increase q from 5 to 30 (ablation) | LoKI on LB Reranker · 6 general benchmarks | avg degradation (%): 6.12% -> 0.46% | Mixed |
| add LoRA (r=32) on top of LoKI q=30 (ablation) | LoKI (q=30) · ToolACE, BFCL overall and forgetting | overall accuracy / avg degradation: 58.93 -> 57.16; 1.23% -> 1.26%; params 563M -> 16M | No clear effect |
| use LoKI instead of full fine-tuning | Qwen2.5-0.5B-Instruct on LB Reranker · 6 general benchmarks | avg degradation (%): 84.13% -> 0.46% | Improved |
Closest papers by meaning
- Interweaving Memories of a Siamese Large Language Model (AAAI 2025 · similarity 0.89)
- Understanding Catastrophic Forgetting in Language Models via Implicit Inference (ICLR 2024 · similarity 0.88)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.88)
- Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis (ICML 2025 · similarity 0.88)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.87)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
