Research map / Large language models / Instruction and fine-tuning

Alignment Dynamics in LLM Fine-Tuning

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives a closed-form, first-order update for an alignment score under SFT (via an empirical NTK with a relatively-stable-kernel assumption), decomposing it into a Rebound Force and a Driving Force, and validates predicted rebound-under-reversal, narrowness dependence, and Rehearsal Priming on safety, emergent misalignment, and IMDb sentiment.

Problem

Alignment of LLMs is fragile under subsequent fine-tuning, and existing explanations are either gradient-geometric or static distributional descriptions that do not link parameter-space learning dynamics to function-space alignment behavior.

Approach

Defines an alignment score as the probability of aligned completions and derives its first-order change under fine-tuning via the eNTK, yielding a Rebound Force (self-interaction, scaled by alignment state and distribution narrowness) and a Driving Force (training distribution contrasted against aligned vs non-aligned posteriors), and predicts Rehearsal Priming from latent posterior residue.

Key results

• Safety-aligned models rebound rapidly to baseline under reverse SFT on unsafe data regardless of Stage 1 duration.
• Narrower safety training distributions (Fixed Template) lead to lower Stage 1 convergence and faster Stage 2 safety degradation than more diverse ones.
• Deeper Stage 1 exposure yields faster Stage 3 re-alignment across BeaverTails, emergent misalignment, and IMDb settings (Rehearsal Priming).

Surprising findings

• Misaligned states are also rapidly reversed: a few benign SFT steps drop the misalignment rate precipitously.
• Models realigned by benign fine-tuning can have broad misalignment reactivated by only a few malicious steps.

Limitations

• Assumes a relatively stable eNTK whose validity at industrial scale is unclear.
• Main results restricted to SFT; DPO and RLHF extensions left to future work.
• Experiments use small-scale (8,192-sample) sequential SFT on base models with a single learning rate.

Experimental setup

Datasets: BeaverTails HEx-PHI Alpaca-cleaned IMDb Emergent Misalignment dataset

Benchmarks: BeaverTails HEx-PHI Emergent Misalignment 44-prompt elicitation set IMDb prompts

Metrics: safety rate misalignment rate positive score

Baselines: Stage 1 forward fine-tuning (initial alignment)

Builds on: empirical NTK for LLM fine-tuning Superficial Alignment Hypothesis Rebound Effect

Closest papers by meaning

• The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety (NeurIPS 2026 · similarity 0.92)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.91)
• Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.90)
• Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment (ICLR 2024 · similarity 0.90)
• LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations (NeurIPS 2026 · similarity 0.90)
• Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.90)