Research map / Learning theory and algorithms / Kernels and regression theory
A Theoretical Analysis of Fine-tuning with Linear Teachers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the covariance-aware bound instead of the source-target distance bound | Linear-regression fine-tuning risk prediction · MNIST transfer tasks, 10–30 samples | R^2: 0.69 ± 0.03, 0.68 ± 0.04, 0.66 ± 0.04, 0.64 ± 0.03, 0.62 ± 0.02 -> 0.86 ± 0.02, 0.89 ± 0.02, 0.84 ± 0.02, 0.75 ± 0.01, 0 | Helped |
| Use the covariance-aware bound instead of the adapted prior bound | Linear-regression fine-tuning risk prediction · MNIST transfer tasks, 10–30 samples | R^2: 0.73 ± 0.03, 0.75 ± 0.03, 0.74 ± 0.03, 0.71 ± 0.02, 0.67 ± 0.02 -> 0.86 ± 0.02, 0.89 ± 0.02, 0.84 ± 0.02, 0.75 ± 0.01, 0 | Helped |
| Align the source-target difference with the top covariance eigenspace rather than the bottom eigenspace (ablation) | Linear-regression fine-tuning · Designed covariance experiment; very few target samples | Population loss | Helped |
| Align the source-target difference with the bottom covariance eigenspace rather than the top eigenspace (ablation) | Linear-regression fine-tuning · Designed covariance experiment; larger target sample counts | Population loss | Helped |
| Increase depth when source and target task scales differ (ablation) | Deep linear fine-tuning · Synthetic tasks with scaled, noisy source-target relation | Accuracy | Helped |
| Increase target-task norm rather than source-task norm (ablation) | Deep linear fine-tuning · Synthetic scaling-factor experiment | Generalization | Hurt |
| Freeze the first layer after pretraining rather than train all layers (ablation) | Deep linear fine-tuning · Target task in the two-layer linear-model comparison | Generalization | Hurt |
| Use a deep linear network rather than a single-layer linear predictor when source and target directions match but norms differ (ablation) | Fine-tuning with aligned source and target teachers · Aligned tasks with different norms | Population risk: 0 -> ((alpha-1)/alpha)^2||Sigma^(1/2)P_perp theta_T||^2 | Helped |
Closest papers by meaning
- Sample complexity of data-driven tuning of model hyperparameters in neural networks with structured parameter-dependent dual function (NeurIPS 2025 · similarity 0.87)
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.87)
- On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm (ICML 2024 · similarity 0.87)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.87)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.86)
- Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.