Research map / Large language models / Instruction and fine-tuning
An Emulator for Fine-tuning Large Language Models using Small Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces emulated fine-tuning (EFT), which reweights a base model's conditionals by the log-ratio of a fine-tuned and base model to decouple pre-training and fine-tuning scale, and proposes up-scaling to approximate fine-tuning large models using small fine-tuned models.
Problem
Understanding which capabilities come from pre-training versus fine-tuning and at what scale, and how to get the benefits of fine-tuning a large model without the compute cost of fine-tuning it.
Approach
Treat the fine-tuned-minus-base log-probability difference as an implicit reward from KL-constrained RL, and apply it as a importance weight on a different-sized base model's conditionals at sampling time, so pre-training and fine-tuning scales can be chosen independently without any training.
Key results
- Up-scaling Llama-2-7B-chat with Llama-2-70B base recovers nearly 70% of the factuality gains of the large fine-tuned model relative to the small fine-tuned model alone (normalized, averaged over families and datasets).
- Speculative EFT decoding accelerates sampling from the up-scaled 7B-to-70B model by nearly 2.5x (6.0 to 13.8 tokens/sec on HH), producing identical samples.
- Human and GPT-4 factuality labels agree 61% of the time; on disagreements, gold labels find GPT-4 correct 77% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale pre-training while holding fine-tuning scale fixed | EFT across Llama-1, Llama-2, and Falcon · Anthropic-HH and ELI5 | factuality | Helped |
| Scale fine-tuning while holding pre-training scale fixed | EFT across Llama-1, Llama-2, and Falcon · Anthropic-HH and ELI5 | helpfulness | Helped |
| Combine a large base model with a small fine-tuned model (up-scaling) | EFT up-scaling · Anthropic-HH and ELI5 | factuality: nearly 70% | Helped |
| Combine a large base model with a small fine-tuned model (up-scaling) | EFT up-scaling · Anthropic-HH and ELI5 | helpfulness | Helped |
| Use speculative decoding with block size 8 instead of no speculative decoding (ablation) | EFT up-scaling with Llama-2-70B-base + Llama-2-7B-chat · Anthropic-HH | tokens/sec: 6.0 -> 13.8 | Helped |
| Use speculative decoding with block size 8 instead of no speculative decoding (ablation) | EFT up-scaling with Llama-2-70B-base + Llama-2-7B-chat · ELI5 | tokens/sec: 6.1 -> 15.1 | Helped |
| Apply top-p 0.95 filtering to up-scaling weights rather than no truncation (ablation) | Llama-2 up-scaling · Llama-2 up-scaling | factual errors/prompt: 0.300 -> 0.289 | Helped |
| Apply top-p 0.95 filtering to up-scaling weights rather than no truncation (ablation) | Llama-2 up-scaling · Llama-2 up-scaling | helpfulness: 66.8 -> 67.0 | Helped |
| Apply top-p 0.9 or 0.8 filtering to up-scaling weights rather than no truncation (ablation) | Llama-2 up-scaling · Llama-2 up-scaling | factual errors/prompt: 0.300 -> 0.352; 0.300 -> 0.348 | Hurt |
| Apply top-p 0.9 or 0.8 filtering to up-scaling weights rather than no truncation (ablation) | Llama-2 up-scaling · Llama-2 up-scaling | helpfulness: 66.8 -> 67.2; 66.8 -> 67.0 | Helped |
| Use speculative block size 16 instead of block size 8 (ablation) | EFT up-scaling with Llama-2-70B-base + Llama-2-7B-chat · Anthropic-HH | tokens/sec: 13.8 -> 12.1 | Hurt |
| Use speculative block size 16 instead of block size 8 (ablation) | EFT up-scaling with Llama-2-70B-base + Llama-2-7B-chat · ELI5 | tokens/sec: 15.1 -> 14.2 | Hurt |
Surprising findings
- Untuned large base model without truncation performs relatively strongly on factual errors, limiting the benefit of conservative top-p reweighting.
- GPT-4 was more accurate than time-limited human crowdworkers on factuality disagreements.
Limitations
- Relies on GPT-4 as proxy for human evaluation; human agreement is only 61%.
- Falcon-180B used in 8-bit quantization for down-scaling experiments, which may affect generation quality.
- Up-scaling ratios can be extremely large for low-probability tokens, leading to low-quality high-probability tokens.
Experimental setup
Datasets: Anthropic Helpful-Harmless ELI5
Benchmarks: Anthropic Helpful-Harmless prompts ELI5 prompts
Metrics: GPT-4 helpfulness probability GPT-4 factual error count GPT-4 harmlessness score tokens per second
Baselines: Llama-2-chat 7B Llama-2-chat 70B Vicuna Falcon-instruct DPO
Builds on: KL-constrained RLHF DPO speculative decoding contrastive decoding
Closest papers by meaning
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.91)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.90)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.90)
- Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.89)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.89)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.