Research map / Large language models / Instruction and fine-tuning

Pretraining Curricula Enable Selective Fine-tuning

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that imbalanced (sequential) pretraining curricula over conflicting copy tasks yield more disentangled, task-specific circuits, reducing overrefusal under refusal fine-tuning and producing more robust rule adherence in a synthetic language task.

Problem

How explicit pretraining curricula affect learning, generalization, circuit entanglement, and the selectivity of downstream fine-tuning (e.g., refusal fine-tuning that should suppress one behavior without overrefusing another).

Approach

Compares imbalanced task-mixture curricula (one task dominant early, annealed to 50:50) against balanced mixtures during pretraining of small transformers, finding imbalanced training induces separable circuits that make refusal fine-tuning more selective and rule adherence more robust.

Key results

• Imbalanced curricula yielded 100/100 seeds reaching generalizing solutions vs 22/100 for balanced curricula on the copy-first/copy-last ICL task.
• Under balanced training, ablating L2 attention drops copy-first and copy-last to 51.9% and 42.9%; under imbalanced training, copy-last is preserved at 99.1% after L2 attention ablation.
• In SLL, imbalanced curricula yield lower P(Misaligned) on elicitation prompts after fine-tuning, while free generations look similar across curricula.

Surprising findings

• Balanced curricula with infinite fresh data failed to produce generalizing solutions (0/20 seeds), while imbalanced curricula succeeded (19/20).
• Constant imbalanced mixtures (15:85, 85:15) without fading also produced generalizing solutions, suggesting the dynamic resampling is not the driver.

Limitations

• Experiments use tiny transformers and simplified synthetic tasks, limiting relevance to LLMs.
• Analyses restricted to seeds reaching >=90% accuracy, with small seed counts (e.g., 22/100 for balanced).

Experimental setup

Datasets: synthetic copy-first / copy-last ICL sequences synthetic biographical profile language (SLL)

Benchmarks: held-out copy-first/copy-last ICL test set (500 sequences) SLL free generations and elicitation prompts

Metrics: test accuracy refusal rate overrefusal rate mean-ablation accuracy direct logit attribution activation patching recovery P(Misaligned)

Baselines: balanced task mixture curriculum

Builds on: mean ablation direct logit attribution denoising activation patching AdamW

Closest papers by meaning

• Understanding Catastrophic Forgetting in Language Models via Implicit Inference (ICLR 2024 · similarity 0.87)
• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (ICLR 2024 · similarity 0.86)
• The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety (NeurIPS 2026 · similarity 0.86)
• Learning Versatile Skills with Curriculum Masking (NeurIPS 2024 · similarity 0.85)
• An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.84)
• On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm (ICML 2024 · similarity 0.84)