Research map / Large language models / Instruction and fine-tuning
Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mechanistic study on synthetic tasks (Tracr-compiled transformers and PCFG-trained minGPT) plus TinyStories shows fine-tuning mostly learns localized wrappers over pretrained capabilities rather than altering them, and these can be revived by reverse fine-tuning.
Problem
Understanding how fine-tuning alters a pretrained model's underlying capabilities: does it create novel capabilities, delete them, or only modulate existing ones, and how pernicious is this for safety?
Approach
Defines capabilities and their relevance to fine-tuning tasks, then uses controlled synthetic settings with mechanistic tools (pruning, attention maps, linear probes) plus a proposed reverse fine-tuning (reFT) protocol to show fine-tuning typically learns a minimal, localized wrapper over the existing capability, which can be revived sample-efficiently.
Key results
- Pruning as few as 5-15 neurons in PCFG models restores pretraining-task accuracy when a small learning rate is used for fine-tuning.
- Linear probes recover pretraining-task counts from fine-tuned models, especially at small learning rates, indicating the underlying capability persists.
- Reverse fine-tuning revives deleted/wrapped capabilities in 30-300 iterations on TinyStories, versus a baseline that never reaches the same loss; e.g., the twist proportion reaches 81% at 300 iterations for F (eta_S) vs 44% for the control at 300 iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reverse fine-tune after Filtering (η_M) to generate stories with twists | TinyStories-Instruct models fine-tuned to avoid twists · TinyStories-Instruct | twist proportion at iteration 30: 81% vs 31% | Helped |
| Use Filtering + Randomisation (η_M), then reverse fine-tune (ablation) | TinyStories-Instruct models fine-tuned to avoid twists · TinyStories-Instruct | twist proportion at iteration 30: 56% vs 31% | Helped |
| Use Filtering + Mix & Match (η_M), then reverse fine-tune (ablation) | TinyStories-Instruct models fine-tuned to avoid twists · TinyStories-Instruct | twist proportion at iteration 30: 88% vs 31% | Helped |
| Use a small learning rate (η_S) for Filtering, then reverse fine-tune (ablation) | TinyStories-Instruct models fine-tuned to avoid twists · TinyStories-Instruct | twist proportion at iteration 30: 88% vs 31% | Helped |
| Use Filtering + Randomisation with a small learning rate (η_S), then reverse fine-tune (ablation) | TinyStories-Instruct models fine-tuned to avoid twists · TinyStories-Instruct | twist proportion at iteration 30: 44% vs 31% | Helped |
| Use Filtering + Mix & Match with a small learning rate (η_S), then reverse fine-tune (ablation) | TinyStories-Instruct models fine-tuned to avoid twists · TinyStories-Instruct | twist proportion at iteration 30: 81% vs 31% | Helped |
| Prune a small number of neurons from the fine-tuned model (ablation) | PCFG Counter model · Pretraining task: count O_PT | accuracy | Helped |
| Increase the pretraining sampling probability of the fine-tuning target token (ablation) | PCFG Counter model · Fine-tuning test data with and without spurious correlation | accuracy | Helped |
| Use a larger fine-tuning learning rate when the pretraining capability is not weakly relevant (ablation) | Tracr Counter model · Tracr Counter | attention to the pretraining target | Hurt |
| Fine-tune at a lower learning rate (ablation) | TinyStories-Instruct models fine-tuned to avoid twists · TinyStories-Instruct | probe accuracy for the twist feature | No clear effect |
| Reverse fine-tune on data from the original pretraining distribution | PCFG and Tracr fine-tuned models · Pretraining tasks | accuracy | Helped |
Surprising findings
- Fine-tuning with a forbidden-feature objective on TinyStories leaves the feature linearly probeable even when the fine-tuning removes its behavior.
- Models whose pretraining already gave high sampling probability to the target token avoid spurious correlations, the opposite of what might be expected from a simple fine-tuning view.
Limitations
- Experiments are on small synthetic setups (Tracr models, ~3M-parameter minGPT) and may not transfer to large LLMs.
- Capability definition is idealized; not all capabilities in real pretrained models act as cleanly as the definition requires.
- TinyStories analysis uses a single forbidden feature (twist) and a GPT-3.5 classifier for evaluation.
Experimental setup
Datasets: PCFG-generated synthetic strings TinyStories-Instruct
Benchmarks: synthetic Counter task (PCFG) Max-identifier and Counter (Tracr) TinyStories twist feature probing and reFT
Metrics: accuracy linear probe accuracy loss during reverse fine-tuning percent of stories with twist
Baselines: Scr.+FT (scratch-pretrained-then-finetuned control) Not in PT control model
Builds on: Tracr PCFG pretraining setup minGPT
Closest papers by meaning
- Understanding Catastrophic Forgetting in Language Models via Implicit Inference (ICLR 2024 · similarity 0.89)
- Fine-Tuning is Fine, if Calibrated (NeurIPS 2024 · similarity 0.88)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.88)
- TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs (ICML 2025 · similarity 0.88)
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.88)
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.