Research map / Large language models / Retrieval and knowledge
Understanding LoRA as Knowledge Memory: An Empirical Analysis
Seungju Back, Dongwoo Lee, Naun Kang, Taehee Lee, S. K. Hong, Youngjune Gwon, Sungjin Ahn
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train LoRA on synthetic QA instead of raw text (ablation) | LoRA memory, Llama-3.1-8B · PaperQA | LLM-judge score: 3.187 -> 5.893 | Improved |
| mix Original, Summary8, Rewrite4 and QA40 instead of QA40 alone (ablation) | LoRA memory, Llama-3.1-8B · PaperQA | LLM-judge score: 5.893 -> 6.822 | Improved |
| mix Summary8 with QA40 instead of QA40 alone (ablation) | LoRA memory, Qwen3-8B · PaperQA | LLM-judge score: 5.409 -> 5.662 | Improved |
| scale base model from 1.7B to 8B (ablation) | LoRA memory, Qwen3 · PaperQA | LLM-judge score | No clear effect |
| generate synthetic data with GPT-4.1 instead of Llama-3.1-8B (ablation) | LoRA memory · PaperQA | LLM-judge score | Improved |
| increase LoRA rank (ablation) | single LoRA · CounterFact | efficacy | Improved |
| use highest rank instead of low rank (ablation) | single LoRA · PhoneBook and CounterFact | parameter efficiency | Hurt |
| partition knowledge across multiple small LoRAs with oracle routing (ablation) | multi-LoRA · PhoneBook 64K | exact match | Improved |
| replace oracle routing with embedding-based routing (ablation) | multi-LoRA · PaperQA | LLM-judge score | Hurt |
| merge LoRAs with TIES instead of DARE or linear (ablation) | multi-LoRA merging · PaperQA | LLM-judge score | Improved |
| rescale concatenation by 1/sqrt(N) (ablation) | multi-LoRA merging (CAT) · PaperQA | LLM-judge score | Improved |
| merge more LoRAs (N from 1 to 5) with TIES under ground-truth selection (ablation) | multi-LoRA · PaperQA | LLM-judge score | Hurt |
Closest papers by meaning
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.90)
- Doc-to-LoRA: Learning to Instantly Internalize Contexts (ICML 2026 · similarity 0.90)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.89)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.89)
- LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
- LoRA-Gen: Specializing Large Language Model via Online LoRA Generation (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
