Research map / Large language models / Instruction and fine-tuning
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KaSA, a PEFT method that truncates minor singular components of the base weights and adapts knowledge-aware singular values in an SVD-form update, evaluated on GLUE, E2E NLG, instruction following, and commonsense reasoning.
Problem
LoRA and its SVD-based successors (PiSSA, MiLoRA) either update noisy/irrelevant knowledge in the base model or let low-rank updates inherit task-irrelevant knowledge, which hurts fine-tuning performance when adapting LLMs.
Approach
Remove minor singular components of the base weight (noisy/long-tail knowledge) via SVD truncation, then reparameterize the task update in SVD form with learnable knowledge-aware singular values that dynamically activate relevant knowledge, with regularizers keeping the singular vectors near-orthogonal and the singular values small.
Key results
- On GLUE with RoBERTa-base, KaSA averages 86.3 vs 85.2 for LoRA and FFT, using about 0.3M trainable parameters.
- On commonsense reasoning with LLaMA3 8B, KaSA averages 84.6 vs 81.9 for MiLoRA and 80.8 for LoRA (results for baselines taken from prior work).
- Instruction following on MT-Bench: KaSA reaches 4.97 on Gemma 7B and 4.95 on LLaMA2 13B, above FFT and the PEFT baselines reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KaSA instead of FFT | RoBERTa-base · GLUE average | average: 85.2 -> 86.3 | Helped |
| Use KaSA instead of FFT | RoBERTa-large · GLUE average | average: 88.2 -> 89.0 | Helped |
| Use KaSA instead of the strongest listed PEFT baseline | Gemma 7B · MT-Bench | score: 4.66 -> 4.97 | Helped |
| Use KaSA instead of FFT | Mistral 7B · Classification | score: 6.73 -> 5.72 | Hurt |
| Use KaSA instead of FFT | Mistral 7B · MT-Bench | score: 4.22 -> 4.58 | Helped |
| Use KaSA instead of FFT | LLaMA3 8B · Closed QA | score: 8.90 -> 6.81 | Hurt |
| Use KaSA instead of MiLoRA | LLaMA3 8B · MT-Bench | score: 4.32 -> 4.71 | Helped |
| Use KaSA instead of MiLoRA | LLaMA2 7B · Commonsense170K | average: 79.2 -> 81.5 | Helped |
| Use KaSA instead of MiLoRA | LLaMA3 8B · Commonsense170K | average: 81.9 -> 84.6 | Helped |
| Add knowledge-based SVD truncation (ablation) | KaSA on RoBERTa-base · MRPC, CoLA, and RTE | task performance: 2.05% to 3.25% | Helped |
| Add knowledge-aware singular-value adaptation (ablation) | KaSA on RoBERTa-base · MRPC, CoLA, and RTE | task performance: 2.05% to 3.25% | Helped |
| Add singular-value regularization L2 (ablation) | KaSA on RoBERTa-base · MRPC, CoLA, and RTE | task performance: 2.05% to 3.25% | Helped |
Surprising findings
- Standard FFT is not always an upper bound: KaSA outperforms FFT in 13 of 14 GLUE RoBERTa settings, and in the GPT-4o-judged instruction tasks on several models FFT scores lower than KaSA.
Limitations
- Relies on LLM-as-judge (GPT-4o/GPT-4) evaluation for instruction following and MT-Bench, without human evaluation.
- Hyperparameters (beta, gamma, rank) were tuned per dataset on GLUE, and commonsense experiments used fixed configs; sensitivity to these choices is not fully characterized.
Experimental setup
Datasets: GLUE E2E NLG Challenge No Robots (seed) Alpaca Commonsense170K MT-Bench
Benchmarks: GLUE E2E NLG Challenge MT-Bench BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Metrics: accuracy Matthews correlation Pearson correlation BLEU NIST METEOR ROUGE-L CIDEr GPT-4o judge score (1-10) MT-Bench score
Baselines: FFT Adapter-H Adapter-D Adapter-L Adapter-P LoRA DyLoRA VeRA DoRA AdaLoRA PiSSA MiLoRA
Builds on: LoRA PiSSA MiLoRA AdaLoRA CorDA
Closest papers by meaning
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.90)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.89)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.89)
- LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.89)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.