Research map / Large language models / Instruction and fine-tuning

Model Spec Midtraining: Improving How Alignment Training Generalizes

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes model spec midtraining (MSM), which fine-tunes base models on synthetic documents discussing a Model Spec before standard alignment fine-tuning, to control and improve how alignment data generalizes, including reducing agentic misalignment.

Problem

Standard alignment fine-tuning on spec-aligned demonstrations can yield shallow alignment that generalizes poorly OOD, since demonstration data underspecifies the intended generalization (e.g., agentic misalignment).

Approach

Insert a midtraining phase between pre-training and alignment fine-tuning in which the model trains next-token prediction on diverse synthetic documents that discuss the content and rationale of a Model Spec, so that identical downstream demonstrations generalize according to the spec.

Key results

• On agentic misalignment, MSM + AFT reduces misalignment on Qwen3-32B from 54% to 7%, beating deliberative alignment (14%); Qwen2.5-32B goes from 68% to 5% (vs 48% for deliberative alignment).
• The same cheese AFT data generalizes to pro-America or pro-affordability values depending on which spec was midtrained.
• MSM makes AFT roughly 10x to 60x more token-efficient on agentic misalignment, depending on model and CoT setting.

Surprising findings

• MSM + AFT without CoT outperforms AFT with CoT on both Qwen models, reducing reliance on CoT supervision.
• Models midtrained on a Claude or human-describing spec perform similarly to Qwen-described MSM, suggesting attributed identity matters little.

Limitations

• Only tests one misalignment form (instrumental unilateral harmful action); not reward hacking or sycophancy.
• Not tested against stronger misalignment pressure such as RL or high-compute post-training.
• Only SFT-based alignment tested; no RL stacking.

Experimental setup

Datasets: No Robots Tulu3 IF NuminaMath CoT Self-Oss-Instruct Smol-constraints APIGen-Function-Calling Smol-summarize LIMA LongAlign

Benchmarks: Agentic misalignment (27 evals) open-ended QA spec alignment (151 questions) held-out value preference pairs

Metrics: misalignment rate spec alignment score (1-10) value-aligned preference rate

Baselines: AFT (with CoT) AFT (no CoT) deliberative alignment

Builds on: synthetic document finetuning deliberative alignment LoRA

Closest papers by meaning

• LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations (NeurIPS 2026 · similarity 0.89)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.89)
• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.89)
• Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.89)
• Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.89)
• Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment (ICLR 2024 · similarity 0.89)