Research map / Architectures and efficiency / Pruning and efficient tuning
Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why linear probing then fine-tuning (LP-FT) works for Transformer language models using the empirical NTK, showing the classifier norm grows during LP and reduces feature change, and that temperature scaling fixes the calibration cost.
Problem
Explaining the training dynamics and success of LP-FT, which prior analysis covered only for two-layer linear models, when applied to complex Transformer-based language models and LoRA.
Approach
Decomposes the empirical NTK of a classifier into a pre-train-effective (diagonal) and an FT-effective component, showing the classifier weight norm after LP scales the FT-effective part and thus reduces feature distortion; CE training inflates this norm, which hurts calibration but can be corrected by temperature scaling; extends the analysis to LoRA as a scalar rescaling of the FT NTK.
Key results
- On RTE, LP increases the classifier norm from 0.776 (pre-trained) to 31.0, while FT yields 0.784 and LoRA 1.21.
- Temperature scaling on RTE reduces MCE for LP-FT from 63.95% to 13.94% (improvement 50.01 points, the largest among methods).
- On CB, LP-FT's FT-effective NTK component has kernel regression test accuracy 86.31%, and LoRA's FT-effective component has rank 500 vs FT's 608, with normalized singular value distributions closely matching, suggesting LoRA's FT NTK differs only by a scalar.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LP-FT instead of standard FT | RoBERTa-base · CB | feature cosine similarity: 0.336 -> 0.804 | Helped |
| Use LP-FT instead of standard FT | RoBERTa-base · RTE | feature difference norm: 2.16 \times 10^{1} -> 4.70 \times 10^{0} | Helped |
| Use LP-FT instead of standard FT | RoBERTa-base · CB kernel regression | test accuracy: 79.76 -> 86.31 | Helped |
| Use LP-FT instead of LoRA | RoBERTa-base · RTE | feature difference norm: 4.70 \times 10^{0} -> 4.63 | No clear effect |
| Apply temperature scaling | LP-FT · RTE test | ECE (%): 21.72 -> 5.48 | Helped |
| Apply temperature scaling | LP-FT · RTE test | MCE (%): 63.95 -> 13.94 | Helped |
| Apply temperature scaling | FT · RTE test | ECE (%): 21.16 -> 5.13 | Helped |
| Increase classifier weight norm at the start of FT (ablation) | LP-FT and FT · training feature changes | feature difference | Helped |
Surprising findings
- Increased classifier norm during LP reduces feature changes in LP-FT, adding a norm-based explanation beyond the near-optimal head explanation.
- Larger classifier norms worsen calibration, with LP-FT showing the largest MCE gains from temperature scaling.
Limitations
- Analysis is restricted to the NTK regime and a one-epoch gradient descent approximation of fine-tuning.
- Experiments are limited to natural language processing with a single RoBERTa-base model.
Experimental setup
Datasets: BoolQ CB RTE WiC WSC CoLA MRPC SST-2 Amazon Reviews Dynasent SemEval SST-5
Benchmarks: SuperGLUE GLUE BOSS PubMed 20k RCT
Metrics: accuracy cosine similarity difference norm Fisher's discriminant ratio ECE MCE kernel regression accuracy Frobenius norm rank
Baselines: FT LP-FT LoRA LP-LoRA LP
Builds on: linear probing then fine-tuning (LP-FT) neural tangent kernel empirical NTK LoRA temperature scaling
Closest papers by meaning
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.91)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.87)
- LoRA Training in the NTK Regime has No Spurious Local Minima (ICML 2024 · similarity 0.87)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.87)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.87)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.