Research map / Large language models / Instruction and fine-tuning
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LISA, a memory-efficient LLM fine-tuning method that randomly unfreezes a few intermediate layers per sampling period (embedding and LM head always trained), motivated by skewed layerwise weight norms observed in LoRA.
Problem
LoRA's low-rank restriction limits its performance relative to full-parameter fine-tuning in large-scale LLM fine-tuning and continual pre-training, while full training is memory-prohibitive.
Approach
Observes that LoRA's layerwise weight norms are skewed (embedding, LM head, and bottom/top layers dominate), and uses importance sampling over layers: always update embedding and LM head, and randomly unfreeze a small number of intermediate layers every K steps with AdamW.
Key results
- LISA outperforms LoRA by 10%-35% on MT-Bench for moderate-scale models (e.g., LLaMA-2-7B 4.94 vs LoRA 4.45), with memory comparable to LoRA (LLaMA-2-7B ~21-24G vs LoRA 23-28G).
- On LLaMA-2-70B, LISA reaches MT-Bench 6.72 vs LoRA 6.10 and FT 6.25, and PubMedQA 91.6 vs LoRA 90.8.
- LISA gives about 2.9x speedup over full-parameter training and about 1.5x over LoRA per iteration on LLaMA-2-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LISA instead of LoRA | TinyLlama · MT-Bench | score: 1.90 -> 2.57 | Helped |
| Use LISA instead of LoRA | Mistral-7B · MT-Bench | score: 4.41 -> 4.85 | Helped |
| Use LISA instead of LoRA | LLaMA-2-7B · MT-Bench | score: 4.45 -> 4.94 | Helped |
| Use LISA instead of LoRA | LLaMA-2-70B · MT-Bench | score: 6.10 -> 6.72 | Helped |
| Use LISA instead of full-parameter fine-tuning | LLaMA-2-70B · GSM8K | score: 67.1 -> 61.1 | Hurt |
| Use LISA instead of full-parameter fine-tuning | LLaMA-2-7B continual pre-training · GSM8K | score: 22.21 -> 22.21 | No clear effect |
| Increase sampling layers and adjust learning rate from γ=2, η=5 × 10^-5 to γ=8, η=1.25 × 10^-5 (ablation) | LISA on LLaMA-2-7B · GSM8K | score: 15.77 -> 16.30 | Helped |
| Increase sampling layers and reduce learning rate from γ=2, η=5 × 10^-5 to γ=16, η=6.25 × 10^-6 (ablation) | LISA on LLaMA-2-7B · GSM8K | score: 15.77 -> 16.57 | Helped |
| Use randomly changing sampled layers instead of fixed randomly selected layers (ablation) | LISA on LLaMA-2-7B · MT-Bench | score: 4.62 -> 4.94 | Helped |
| Use randomly changing sampled layers instead of fixed randomly selected layers (ablation) | LISA on LLaMA-2-7B · MT-Bench | score: 4.60 -> 4.94 | Helped |
| Use randomly changing sampled layers instead of fixed randomly selected layers (ablation) | LISA on LLaMA-2-7B · MT-Bench | score: 4.67 -> 4.94 | Helped |
Surprising findings
- LISA outperforms full-parameter training on several instruction-following and continual pre-training results, which the authors attribute to implicit regularization from restricting unfrozen layers.
- LoRA's layerwise weight norms are skewed with embedding and LM head layers having norms hundreds of times larger than intermediate layers.
Limitations
- Forward pass still requires the full model in memory, so memory savings are bounded as with LoRA; quantization (QLoRA) combination left for future work.
- Uniform random sampling of intermediate layers is likely not the optimal importance sampling strategy.
Experimental setup
Datasets: Alpaca-GPT4 GSM8K PubMedQA OpenWebMath Alpaca
Benchmarks: MT-Bench MMLU AGIEval WinoGrande GSM8K PubMedQA
Metrics: MT-Bench score accuracy peak GPU memory seconds per iteration
Baselines: Full parameter training LoRA GaLore Vanilla (no fine-tuning)
Builds on: LoRA AdamW GaLore DoLa
Closest papers by meaning
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.90)
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.90)
- MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling (NeurIPS 2025 · similarity 0.90)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.90)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.