Research map / Large language models / Instruction and fine-tuning
Selecting Large Language Model to Fine-tune via Rectified Scaling Law
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Rectified Scaling Law with a pre-learned data size term that fits the two-phase fine-tuning loss curve, and an Accept-then-Stop algorithm that uses it to select the best LLM to fine-tune at hundreds of times lower cost.
Problem
Selecting the best pre-trained LLM to fine-tune under resource constraints, without fully fine-tuning every candidate; existing scaling laws fail to capture the fine-tuning loss curve, which has a pre-power phase at small data sizes.
Approach
Discovers a 'pre-power phase' in low-data fine-tuning scaling curves and adds a pre-learned data size term D_l to the scaling law (B/(D_l+D^beta)+E), then uses a stop-when-phase-changes rule to extrapolate the power-phase line from small subsets to the full dataset for model ranking.
Key results
- AtS achieves average PearCorr of 93.8% on Gigaword and 84.6% on WMT19 at budget ratio 1/8 avg, versus negative or low correlations for ZeroShot and ModelSize.
- With only 1/512 of the data, AtS reaches 66.6% average PearCorr versus 38.6% for the second-best method, ZeroShot.
- Rectified law reduces average RMSD to 0.007 from 0.036 for the vanilla scaling law across 30 models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AtS instead of the strongest listed selection baseline, SubTuning | LLM selection · FLAN (average across budget ratios) | PearCorr (%): 62.7 vs 33.5 | Helped |
| Use AtS instead of the strongest listed selection baseline, SubTuning | LLM selection · WMT19 (average across budget ratios) | PearCorr (%): 84.6 vs 63.4 | Helped |
| Use AtS instead of the strongest listed selection baseline, SubTuning | LLM selection · Gigaword (average across budget ratios) | PearCorr (%): 93.8 vs 79.9 | Helped |
| Use the Rectified Scaling Law instead of the vanilla scaling law | fine-tuning loss prediction · Fine-tuning test-loss fit across 30 models and three datasets | RMSD: 0.036 -> 0.007 | Helped |
| Use AtS rather than fitting the Rectified Scaling Law to all observed points (OurFit) | LLM selection · FLAN (γ=1/512) | PearCorr (%): 36.8 -> 45.6 | Helped |
| Use AtS rather than fitting the vanilla law to all observed points (VanillaFit) | LLM selection · FLAN (γ=1/512) | PearCorr (%): 20.7 -> 45.6 | Helped |
| Use AtS rather than fitting the Rectified Scaling Law to all observed points (OurFit) | LLM selection · Gigaword (γ=1/512) | PearCorr (%): 78.5 -> 91.0 | Helped |
| Vary AtS outlier tolerance δ and minimum accepted-pair count k (ablation) | AtS · Gigaword (γ=1/512) | PearCorr (%) | No clear effect |
Surprising findings
- Almost all models enter the power phase once test loss drops below about 2.2, a threshold also reported for emergent abilities.
- ModelSize and zero-shot performance can yield negatively correlated rankings with true fine-tuned performance.
Limitations
- AtS degrades when the data budget ratio is extremely small, since all observed points lie in the pre-power phase.
- Mechanism behind the pre-power phase and phase transition is not fully understood.
- Extension to RLHF, LoRA and other fine-tuning strategies is untested.
Experimental setup
Datasets: FLAN WMT19 En-Zh Gigaword
Benchmarks: FLAN WMT19 En-Zh Gigaword
Metrics: PearCorr RelAcc RMSD
Baselines: ModelSize ZeroShot SubTuning VanillaFit OurFit
Builds on: Scaling Laws for Neural Language Models Broken neural scaling laws Chinchilla
Closest papers by meaning
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.91)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.91)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.91)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.91)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.