Research map / Large language models / Instruction and fine-tuning
LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LensLLM, a PAC-Bayesian-motivated, NTK-based rectified scaling model that predicts fine-tuned LLM test loss from small data subsets to select the best LLM for a downstream task at lower compute cost.
Problem
Efficiently selecting the best open-source LLM for a downstream task without fully fine-tuning all candidates, especially in low-data regimes where fine-tuning loss follows a pre-power phase before the power-law phase.
Approach
Derives a PAC-Bayesian generalization bound with a Hessian term for transformers, then models fine-tuning test loss with a rectified scaling law whose denominator includes an NTK-based test-loss term F(Θ,t), fit on progressively halved data subsets with a stopping rule to predict full-data performance.
Key results
- On Gigaword, LensLLM achieves PearCorr 85.8% and RelAcc 91.1%, versus 83.5% and 89.2% for the best baseline (Rectified Scaling Law).
- Reports up to 88.5% compute reduction versus FullTuning; LensLLM costs of 0.48, 0.59, and 0.97 x10^21 FLOPs across FLAN, Wikitext, Gigaword.
- Test loss RMSE on Wikitext is about 0.026 vs 0.132 for Rectified Scaling Law on OPT-6.7B (reported in text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LensLLM instead of Rectified Scaling Law | LLM selection · FLAN | PearCorr (%): 75.2 -> 78.4 | Helped |
| Use LensLLM instead of Rectified Scaling Law | LLM selection · Wikitext | RelAcc (%): 88.8 -> 90.2 | Helped |
| Use LensLLM instead of Rectified Scaling Law | LLM selection · Gigaword | PearCorr (%): 83.5 -> 85.8 | Helped |
| Use LensLLM instead of Rectified Scaling Law | test-loss prediction on OPT-6.7B · Wikitext | RMSE (× 10⁻¹): 1.32 -> 0.26 | Helped |
| Use progressive LensLLM sampling instead of FullTuning | LLM selection · FLAN | computational cost (10²¹ FLOPs): 3.35 -> 0.48 | Helped |
| Vary stopping criteria γ and τ (ablation) | LensLLM · FLAN, Gigaword, and Wikitext | PearCorr: 78.31 to 78.43; 85.66 to 85.79; 82.42 to 82.61 | No clear effect |
| Vary batch size and learning rate (ablation) | LensLLM · FLAN, Gigaword, and Wikitext | Pearson correlation: 78.32 to 78.43; 85.69 to 85.79; 82.50 to 82.61 | No clear effect |
Surprising findings
- ZeroShot and ModelSize yield negative Pearson correlations on FLAN and Gigaword, indicating they can anti-rank candidate models.
Limitations
- The theoretical assumptions (e.g., Lipschitz bounds, bounded attention, residual gradient lower bound) are strong and loosely justified; the proofs contain informal steps such as assuming the model fits the data and Hessian-trace scaling derived heuristically.
- Evaluation covers only small-to-mid-sized encoder-decoder and decoder models on three benchmarks with a limited candidate pool.
- Ablations show only small sensitivity to thresholds and hyperparameters, and no evaluation on modern instruction-tuned LLMs is reported.
Experimental setup
Datasets: FLAN Wikitext Gigaword
Benchmarks: FLAN Wikitext Gigaword
Metrics: Pearson correlation (PearCorr) Relative Accuracy (RelAcc) RMSE FLOPs
Baselines: Rectified Scaling Law NLPmetrics SubTuning ZeroShot ModelSize FullTuning
Builds on: Rectified Scaling Law Neural tangent kernel PAC-Bayesian generalization bound LogME
Closest papers by meaning
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.91)
- Selecting Large Language Model to Fine-tune via Rectified Scaling Law (ICML 2024 · similarity 0.91)
- One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.91)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.90)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.