Research map / Large language models / Instruction and fine-tuning

The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Alignment Instability Condition, a Fisher-geometry theory claiming benign fine-tuning inevitably drifts into alignment-sensitive subspaces with quartic utility loss, supported by a Fisher low-rank check and an overlap score on Qwen3-1.7B and LLaMA-3.2-3B.

Problem

Benign fine-tuning (e.g., math or instruction data) systematically degrades safety guardrails, but existing theories (flat/null-space geometry, data overlap) do not explain when or why this happens.

Approach

Introduces the Alignment Instability Condition: initially orthogonal fine-tuning gradients are bent into high-curvature alignment-sensitive Fisher subspaces by second-order curvature coupling, yielding a quartic-in-time utility loss law and an Overlap Score predictor.

Key results

• Seemingly benign datasets (e.g., Risky Financial Advice) reach harmfulness scores near Pure Bad, e.g., 3.94 vs 4.55 on Qwen3-1.7B LoRA.
• On LLaMA-3.2-3B full fine-tuning, Risky Financial Advice gives HS 3.82 and Pure Bad 4.51, while SamSum gives 1.77.
• Overlap Score separates dangerous from benign datasets for full fine-tuning but not for LoRA, which the authors acknowledge as a limitation.

Surprising findings

• Benign GSM8K random 100 subset gave the highest LoRA harmfulness on Qwen3-1.7B (3.63), above Alpaca and SamSum.
• LoRA shows no clear overlap-score to harmfulness correlation, undermining the framework for LoRA.

Limitations

• Theory relies on idealized assumptions (skill optimality, base model stationary for each skill) and local regularity; the second-order curvature term is not computed in experiments.
• Overlap score uses random projection to 4096 dimensions and first-order approximation; fails to explain LoRA results and Alpaca Top 100.
• Experiments are small (two small models, a single judge, 520 AdvBench queries) and do not directly validate the quartic scaling law.

Experimental setup

Datasets: SamSum Alpaca GSM8K Risky Financial Advice Bad Medical Advice Pure Bad BeaverTails AdvBench

Benchmarks: AdvBench

Metrics: Harmfulness Score (1-5) Overlap Score correlation

Baselines: null-space projection methods flat-geometry null model

Builds on: TRPO PPO Davis-Kahan perturbation theory LoRA emergent misalignment

Closest papers by meaning

• Alignment Dynamics in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.92)
• Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning (ICML 2025 · similarity 0.89)
• Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.88)
• Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates (NeurIPS 2024 · similarity 0.88)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.88)
• Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation (ICLR 2025 · similarity 0.88)