Research map / Continual learning / Continual fine-tuning of LLMs
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CorDA, a PEFT method that runs SVD on the weight multiplied by an input-activation covariance matrix from a few task samples, then initializes a LoRA-style adapter from either the smallest components (knowledge-preserved mode) or largest components (instruction-previewed mode).
Problem
Standard PEFT methods such as LoRA build adapters without regard to the downstream task context or the world knowledge to preserve, leaving a gap to full fine-tuning and causing catastrophic forgetting of pre-trained knowledge.
Approach
Decompose each pre-trained linear weight via SVD of (W·C), where C is the input covariance from a few task samples, and reconstruct with C^-1. The resulting components encode task context; the smallest-r components initialize a trainable adapter while the principal components stay frozen to preserve knowledge, or the largest-r components initialize the adapter to better learn the task.
Key results
- Knowledge-preserved mode (LLaMA-2-7B, math, rank 128) reaches NQ open 9.36 vs 1.91 for LoRA and 1.02 for PiSSA, with average 22.46 vs 20.26 for LoRA.
- Instruction-previewed mode achieves GSM8k 53.90 and MATH 8.52 on LLaMA-2-7B, average 22.55, beating PiSSA (21.46), LoRA (18.30) and DoRA (18.18) and matching or exceeding full fine-tuning average (22.46).
- On GLUE with RoBERTa-base, instruction-previewed CorDA averages 83.38 vs 79.43 for LoRA with 21M parameters, close to full fine-tuning's 83.07.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use instruction-previewed adaptation instead of knowledge-preserved adaptation | CorDA on Math fine-tuning · GSM8k | accuracy: 44.58 -> 53.90 | Helped |
| use instruction-previewed adaptation instead of knowledge-preserved adaptation | CorDA on Math fine-tuning · Math validation | accuracy: 6.92 -> 8.52 | Helped |
| use question-answering context (TriviaQA or NQ open) rather than Wikitext-2 context (ablation) | CorDA KPM · Math fine-tuning | Avg: 21.06 -> 22.53; 21.06 -> 22.46 | Helped |
| use context-oriented decomposition rather than plain SVD with smallest r components (ablation) | CorDA KPM · Math fine-tuning | Avg: 19.93 -> 22.46 | Helped |
| use smallest r components rather than largest r components with plain SVD (ablation) | CorDA KPM · Math fine-tuning | Avg: 21.22 -> 19.93 | Hurt |
| collect covariance context from the test-matched dataset rather than the other dataset (ablation) | CO-SVD reconstruction after discarding the smallest 1024 ranks · Wikitext-2 and PTB | perplexity: 8.69 -> 6.35; 29.69 -> 22.28 | Helped |
| use 32 rather than 256 samples for covariance collection (ablation) | CO-SVD reconstruction after discarding the smallest 1024 ranks · Wikitext-2 and PTB | perplexity: 6.35 -> 6.62; 22.28 -> 22.68 | Hurt |
| collect WizardLM-Evol-Instruct rather than Alpaca context (ablation) | CorDA IPM · MTBench | score: 5.06 -> 5.15 | Helped |
| use instruction-previewed CorDA rather than PiSSA | Math fine-tuning on LLaMA-2-7B · GSM8k | accuracy: 51.48 -> 53.90 | Helped |
| use instruction-previewed CorDA rather than PiSSA | Math fine-tuning on LLaMA-2-7B · Math validation | accuracy: 7.60 -> 8.52 | Helped |
| use instruction-previewed CorDA rather than full fine-tuning | Math, Code, and Instruction Following fine-tuning · HumanEval | score: 25.42 -> 21.03 | Hurt |
| use instruction-previewed CorDA rather than full fine-tuning | Math, Code, and Instruction Following fine-tuning · MBPP | score: 25.64 -> 24.15 | Hurt |
Surprising findings
- Freezing the principal components and adapting only the smallest-r components preserves world knowledge better than adapting the principal ones, even though it gives lower math scores than PiSSA.
- Plain SVD with smallest-r components gives no clear knowledge-retention benefit, indicating that data-context covariance weighting is what drives the effect.
Limitations
- Knowledge-preserved mode trails PiSSA on fine-tuning-task performance (e.g., GSM8k 44.58 vs 51.48).
- Trade-off between preserving knowledge and maximizing task performance is not resolved; combining both modes is left to future work.
- Singular results on 7B-scale LLaMA-2 mainly; reported standard deviations only for a subset of experiments (three seeds in ablation table).
Experimental setup
Datasets: MetaMathQA CodeFeedback WizardLM-Evol-Instruct NQ open TriviaQA WebQS Alpaca Wikitext-2 PTB GLUE
Benchmarks: GSM8k MATH HumanEval MBPP MTBench TriviaQA NQ open WebQS GLUE Wikitext-2 PTB
Metrics: accuracy exact match MTBench score pass@1 perplexity Matthew's correlation Pearson correlation
Baselines: Full fine-tuning LoRA PiSSA DoRA ASVD Plain SVD
Builds on: LoRA PiSSA ASVD
Closest papers by meaning
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.91)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models (ICLR 2025 · similarity 0.89)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.89)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.