Research map / Architectures and efficiency / Pruning and efficient tuning
Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching
Fei Zuo, Yizhou Huang, Kezhi Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses path patching to analyze how LoRA affects transformer internals and proposes layer-sparse adaptation that applies LoRA only to selected layers.
Problem
Standard LoRA applies adaptation uniformly across all transformer layers, ignoring that different layers contribute heterogeneously to fine-tuning.
Approach
Applies mechanistic interpretability (path patching) to study LoRA's effect on model internals, then uses the findings to make layer-wise adaptation sparse rather than uniform.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt only the top 30% layers chosen by frozen-base zero-ablation instead of all layers (rank 16) | LoRA · Commonsense average (8 tasks) | accuracy: 56.9 -> 59.1 | Improved |
| adapt top 30% layers instead of all layers (rank 32) | LoRA · Math average (GSM8K, SVAMP, AQUA) | accuracy: 45.5 -> 47.8 | Improved |
| adapt top 30% layers on larger models (Qwen3-8B) | LoRA · Commonsense average (8 tasks), Qwen3-8B | accuracy: 82.5 -> 82.5 | No clear effect |
| adapt top 30% layers on Qwen3-32B | LoRA · Commonsense average (8 tasks), Qwen3-32B | accuracy: 87.3 -> 87.1 | No clear effect |
| adapt top 30% layers on small models (0.5B to 3B) | LoRA · Commonsense average (8 tasks), r16 | accuracy: +1.2 to +2.2 points (Qwen2.5-3B 66.9 -> 68.8) | Improved |
| use calibrated layer selection instead of random layer selection at 12 of 36 layers (ablation) | Adaptive-LoRA · Alpaca-ZH | perplexity: 16.42 -> 12.32 | Improved |
| use calibrated layer selection instead of sequential first-k layers at high budget (28 of 36) (ablation) | Adaptive-LoRA · Alpaca-ZH | perplexity: 6.55 -> 6.97 | Hurt |
| use calibrated layer selection at the smallest budget (4 of 36 layers) versus sequential (ablation) | Adaptive-LoRA · Alpaca-ZH | perplexity: 21.06 -> 21.05 | No clear effect |
| select all 36 layers (full coverage) with any strategy (ablation) | Adaptive-LoRA · Alpaca-ZH | perplexity: 5.75 for all strategies | No clear effect |
| increase calibration samples from N=10 to N=30 (then 100) (ablation) | Adaptive-LoRA calibration · Qwen-3B selection stability | selection Jaccard: 0.524 -> 0.653 (0.685 at N=100) | Improved |
| add layer selection to rank-8 adapters vs full LoRA on Qwen2.5-3B per-task best scores | LoRA · Qwen2.5-3B 11 tasks (best over ranks) | accuracy gap: +0.8 average gap | Improved |
| compare against continual-learning LoRA methods (TreeLoRA, GainLoRA, InfLoRA) in single-task setting | Adaptive-LoRA · Commonsense average, Qwen2.5-1.5B r16 | accuracy: 58.7 -> 59.1 | Improved |
Experimental setup
Builds on: LoRA path patching
Closest papers by meaning
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.92)
- AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.92)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.91)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.90)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.90)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
