AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

Fei Zuo, Yizhou Huang, Kezhi Wang

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses path patching to analyze how LoRA affects transformer internals and proposes layer-sparse adaptation that applies LoRA only to selected layers.

Problem

Standard LoRA applies adaptation uniformly across all transformer layers, ignoring that different layers contribute heterogeneously to fine-tuning.

Approach

Applies mechanistic interpretability (path patching) to study LoRA's effect on model internals, then uses the findings to make layer-wise adaptation sparse rather than uniform.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
adapt only the top 30% layers chosen by frozen-base zero-ablation instead of all layers (rank 16)LoRA · Commonsense average (8 tasks)accuracy: 56.9 -> 59.1Improved
adapt top 30% layers instead of all layers (rank 32)LoRA · Math average (GSM8K, SVAMP, AQUA)accuracy: 45.5 -> 47.8Improved
adapt top 30% layers on larger models (Qwen3-8B)LoRA · Commonsense average (8 tasks), Qwen3-8Baccuracy: 82.5 -> 82.5No clear effect
adapt top 30% layers on Qwen3-32BLoRA · Commonsense average (8 tasks), Qwen3-32Baccuracy: 87.3 -> 87.1No clear effect
adapt top 30% layers on small models (0.5B to 3B)LoRA · Commonsense average (8 tasks), r16accuracy: +1.2 to +2.2 points (Qwen2.5-3B 66.9 -> 68.8)Improved
use calibrated layer selection instead of random layer selection at 12 of 36 layers (ablation)Adaptive-LoRA · Alpaca-ZHperplexity: 16.42 -> 12.32Improved
use calibrated layer selection instead of sequential first-k layers at high budget (28 of 36) (ablation)Adaptive-LoRA · Alpaca-ZHperplexity: 6.55 -> 6.97Hurt
use calibrated layer selection at the smallest budget (4 of 36 layers) versus sequential (ablation)Adaptive-LoRA · Alpaca-ZHperplexity: 21.06 -> 21.05No clear effect
select all 36 layers (full coverage) with any strategy (ablation)Adaptive-LoRA · Alpaca-ZHperplexity: 5.75 for all strategiesNo clear effect
increase calibration samples from N=10 to N=30 (then 100) (ablation)Adaptive-LoRA calibration · Qwen-3B selection stabilityselection Jaccard: 0.524 -> 0.653 (0.685 at N=100)Improved
add layer selection to rank-8 adapters vs full LoRA on Qwen2.5-3B per-task best scoresLoRA · Qwen2.5-3B 11 tasks (best over ranks)accuracy gap: +0.8 average gapImproved
compare against continual-learning LoRA methods (TreeLoRA, GainLoRA, InfLoRA) in single-task settingAdaptive-LoRA · Commonsense average, Qwen2.5-1.5B r16accuracy: 58.7 -> 59.1Improved

Experimental setup

Builds on: LoRA path patching

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy