Research map / Large language models / Language, safety and interpretability

When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

ICLR 2024 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 28 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically studies scaling of LLM finetuning (full-model tuning, prompt tuning, LoRA) over model size, pretraining data, PET parameter size and finetuning data, proposing a multiplicative joint scaling law on bilingual 1B-16B LLMs for translation and summarization.

Problem

Understanding whether and how scaling factors (LLM model size, pretraining data size, PET parameter size, finetuning data size) affect the performance of different LLM finetuning methods, especially in the data-limited regime.

Approach

Proposes a multiplicative joint scaling law between finetuning data size and each other scaling factor (model size, pretraining data, PET parameters), treating finetuning data as the pivot factor, and compares FMT, prompt tuning and LoRA scaling behavior.

Key results

• Multiplicative joint scaling law generalizes better than additive on held-out data (average held-out error 0.0048 for LLM model size vs 0.0079 additive on WMT14 En-De).
• LLM model size exponent alpha_m often exceeds pretraining data exponent alpha_p across methods and tasks (e.g., FMT WMT En-De alpha_m=0.52 vs alpha_p=0.21).
• PET parameter scaling is weak: alpha_t values are on the order of 1e-3 and sometimes negative (e.g., LoRA on En-De).

Measured effects reported in the paper

Surprising findings

• Increasing PET parameter size (prompt length or LoRA rank) yields marginal or inverse scaling in some settings.
• Full-model tuning often degrades few-shot capability of the base LLM, whereas PET retains it.

Limitations

• Joint scaling law is based on empirical results on closed generation tasks (translation and summarization) without theoretical grounding.
• Pretraining data scaling uses intermediate checkpoints as a proxy, which is sub-optimal.
• Optimization instability for prompt tuning and LoRA and 16B En-Zh pretraining instability reduce extrapolation reliability.

Experimental setup

Datasets: WMT14 English-German WMT19 English-Chinese MLSum CNN/Daily-Mail Flores200

Benchmarks: WMT14 En-De WMT19 En-Zh MLSum Flores200 zero-shot translation

Metrics: token-level perplexity BLEURT RougeL mean absolute deviation

Baselines: full-model tuning prompt tuning LoRA

Builds on: LoRA prompt tuning Adafactor T5X

Closest papers by meaning

• A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.92)
• LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.92)
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.91)
• Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.91)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.91)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.91)