Research map / Large language models / Instruction and fine-tuning
LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIFT, a sparse fine-tuning method that selects the top 5% largest-magnitude weights after a low-rank approximation (Principal Weights) and updates only those, matching LoRA-level memory while beating Full FT on reasoning benchmarks.
Problem
Sparse fine-tuning has lagged in the LLM era because there is no good criterion for identifying which parameters are critical for reasoning; full fine-tuning is costly and prone to overfitting and forgetting on small datasets.
Approach
Weights with the largest magnitude after a rank-r SVD approximation are treated as Principal Weights; a binary mask from these is applied to the original weight matrix so only the selected entries are updated, with optimizer states stored only for them. Plain magnitude selection performs poorly, but becomes effective after rank reduction.
Key results
- On commonsense reasoning with LLaMA-2-7B, LIFT averages 84.66 vs 83.53 for Full FT and 81.25 for LoRA.
- On GPQA Diamond with s1K SFT, LIFT reaches 28.79 on Qwen2.5-1.5B (vs 26.77 Full FT) and 34.85 on Qwen2.5-3B (vs 33.33 Full FT).
- LIFT's optimizer states are about 5% of Full FT's (1.3GB vs 27GB on LLaMA-2-7B), and it retains up to 20% more source-domain accuracy than LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply low-rank approximation before selecting high-magnitude weights, rather than selecting by ordinary weight magnitude, movement score, gradient magnitude, or randomly (ablation) | Sparse fine-tuning on GSM8K · GSM8K | accuracy | Helped |
| Use an intermediate mask update interval instead of an interval that is too small or too large (ablation) | LIFT on LLaMA-2-7B · GSM8K | accuracy | Helped |
| Preserve the largest singular values and singular vectors instead of random, smallest, or hybrid ranks (ablation) | LIFT on LLaMA-2-7B · MATH-10K (7 arithmetic tasks) | mean accuracy | Helped |
| Fine-tune Query and Key modules instead of more adaptable modules such as Value, Up, or Down (ablation) | LIFT on LLaMA-2-7B · MATH-10K (7 arithmetic tasks) | accuracy | Hurt |
| Fine-tune only MLP layers (ablation) | LIFT on LLaMA-2-7B · MATH-10K (7 arithmetic tasks) | average accuracy: 73.74 -> 73.34 | No clear effect |
| Use LoRA rank 32 rather than rank 16, 64, 128, or 256 (ablation) | LIFT on LLaMA-3-8B · Commonsense reasoning (8 tasks) | mean test performance: 86.93 -> 87.88 | Helped |
| Use LoRA rank 128 rather than rank 16, 32, 64, or 256 (ablation) | LIFT on LLaMA-2-7B · Arithmetic reasoning (7 tasks) | mean test performance: 70.91 -> 73.74 | Helped |
| Use LoRA rank 128 rather than rank 16, 32, 64, or 256 (ablation) | LIFT on DeBERTa-v3-base · GLUE | mean test performance: 85.17 -> 89.24 | Helped |
Surprising findings
- Plain magnitude-based sparse fine-tuning performs poorly on LLMs, but becomes highly effective once weights are selected after rank reduction.
- Adding noise at scale 0.01 to LIFT-selected weights drastically degrades perplexity and arithmetic accuracy, while noise on other selections does not.
Limitations
- Uses a single global rank for low-rank approximation across all layers, ignoring differing layer capacities.
- Dynamic mask updates make the total updated parameter count fluctuate, so it is not a fixed-size adapter unless modified.
- Hyperparameters and mask update interval require tuning; reported gains are often small (about 1 point) over Full FT.
Experimental setup
Datasets: s1K Commonsense-170K MATH-10K GLUE Alpaca StrategyQA Wikitext
Benchmarks: GPQA Diamond BoolQ PIQA SIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA MultiArith GSM8K AddSub
Metrics: accuracy pass@1 pass@10 perplexity GLUE score
Baselines: Full FT LoRA DoRA PiSSA S2FT Spectral Adapters SpIEL SIFT
Builds on: LoRA PiSSA S2FT DoRA Adam
Closest papers by meaning
- The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models (NeurIPS 2025 · similarity 0.90)
- UFT: Unifying Supervised and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning (NeurIPS 2025 · similarity 0.87)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.87)
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.87)
- SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.