Research map / Large language models / Instruction and fine-tuning
Linearization Explains Fine-Tuning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that an explicit l2 proximity (weight-decay-to-pretrained) regularizer makes LoRA fine-tuning of RoBERTa-base lie in the NTK/linearized regime, and that the NTK condition number at initialization predicts fine-tuning performance and guides layer selection.
Problem
Understanding why parameter-efficient fine-tuning works and how to choose which layers to tune, given that fine-tuning is not guaranteed to lie in the linearized (NTK) regime.
Approach
Adds an explicit l2 inductive bias toward the pretrained parameters, which provably bounds the gap between the fine-tuned model and its linearization; then uses the NTK's eigenvalues and condition number at initialization to predict fine-tuning performance and perturbation bounds for adding layers.
Key results
- On CoLA and SST-2, increasing the regularization strength lambda (up to 50) reduces the parameter deviation and the KL divergence between the regularized model and its linearization, while accuracy stays roughly unchanged (e.g., CoLA accuracy 79-80% for lambda 0.5-10).
- Across GLUE/IMDb/Yelp LoRA runs with RoBERTa-base, the regularized NTK condition number at initialization correlates with final evaluation accuracy and training loss (e.g., {0,11} with W_q,W_v shows a perfect correlation within layer choice).
- Layer selection guided by NTK eigenvalues at initialization selects {0,7} for CoLA and {0,5,11} for SST-2, consistent with the lowest empirical risk after fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase proximity regularization up to λ=10 rather than removing it (ablation) | LoRA fine-tuning · CoLA and SST-2 | evaluation accuracy: CoLA: 79.67 (λ=0) to 80.24 (λ=1); SST-2: 91.6 (λ=0) to 92.8 (λ=1) | No clear effect |
| overregularize with λ=50 (ablation) | LoRA fine-tuning · CoLA and SST-2 | evaluation accuracy: CoLA: 80.44 (λ=5) -> 74.59; SST-2: 93.0 (λ=0.5) -> 89.3 | Hurt |
| increase regularization strength (ablation) | LoRA fine-tuning · CoLA | parameter distance from pretrained model: 0.960 (λ=0.0) -> 0.280 (λ=50) | Helped |
| increase regularization strength (ablation) | LoRA fine-tuning · SST-2 | parameter distance from pretrained model: 2.519 (λ=0.0) -> 0.292 (λ=50) | Helped |
| add layer 11 to Wq/Wv tuning (ablation) | LoRA Wq/Wv in layer {0} · CoLA | evaluation accuracy: 73.15 -> 73.15 | No clear effect |
| tune Wq/Wv in layers {0,5,11} instead of {0} (ablation) | LoRA Wq/Wv · CoLA | evaluation accuracy: 73.15 -> 73.44 | Helped |
| replace Wq/Wv with Wk in layers {0,5,11} (ablation) | LoRA fine-tuning · CoLA | evaluation accuracy: 73.44 -> 73.25 | Mixed |
| add layer 11 to Wq/Wv tuning (ablation) | LoRA Wq/Wv in layer {0} · SST-2 | evaluation accuracy: 83.48 -> 82.79 | Hurt |
| tune Wq/Wv in layers {0,5,11} instead of {0} (ablation) | LoRA Wq/Wv · SST-2 | evaluation accuracy: 83.48 -> 82.68 | Hurt |
| replace Wq/Wv with Wk in layers {0,5,11} (ablation) | LoRA fine-tuning · SST-2 | evaluation accuracy: 82.68 -> 83.60 | Helped |
| vary Wq/Wv layer selection among {0}, {0,11}, and {0,5,11} (ablation) | LoRA Wq/Wv fine-tuning · Yelp | evaluation accuracy: 88.20; 88.24; 88.21 | No clear effect |
| replace Wq/Wv with Wk in layers {0,5,11} (ablation) | LoRA fine-tuning · Yelp | evaluation accuracy: 88.21 -> 88.23 | No clear effect |
Surprising findings
- The highest NTK condition number does not always correspond to the lowest accuracy when the layer and parameter type change, because sigma is held fixed across configurations.
- Accuracy is largely unaffected by the regularization strength, so the proximity constraint costs little performance.
Limitations
- Theory assumes squared loss and gradient descent, while experiments use cross-entropy and AdamW.
- Experiments are limited to RoBERTa-base with LoRA on classification tasks; decoder models only in appendix.
- The sigma parameter is fixed across configurations, so condition-number comparisons across different parameter sets are not fully fair.
Experimental setup
Datasets: CoLA SST-2 IMDb Yelp
Benchmarks: CoLA SST-2 IMDb Yelp
Metrics: evaluation accuracy evaluation loss NTK condition number KL divergence parameter distance to pretrained model
Baselines: LoRA on W_q and W_v LoRA on W_k
Builds on: neural tangent kernel LoRA Malladi et al. kernel fine-tuning analysis lazy training
Closest papers by meaning
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.91)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.91)
- Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective (NeurIPS 2024 · similarity 0.91)
- Sparse is Enough in Fine-tuning Pre-trained Large Language Models (ICML 2024 · similarity 0.91)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.