Research map / Large language models / Instruction and fine-tuning

Crafting Reversible SFT Behaviors in Large Language Models

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Constructs sparse "carriers" for SFT-induced behaviors via utility-budgeted joint mask and weight-delta optimization (LCDD), then reverses the behavior with a soft-prompt trigger matching base-model activations (SFT-Eraser), without changing weights.

Problem

SFT distributes induced behaviors diffusely across weights, so post-hoc circuit attribution yields correlational rather than causally necessary substructures, limiting selective inference-time control of SFT behaviors.

Approach

Impose sparsity on the SFT weight delta during training via row/column gate masks with an adaptive dual multiplier under a utility budget, so the carrier is the sole locus of the behavior; then a fixed-weight soft trigger that drives carrier write activations toward the base model reverts the behavior.

Key results

• Fixed Response: LCDD keeps 97.5-100% fixed-response rate; trigger drops it to 0.0-1.0% across four models.
• Safety: LCDD keeps WildGuard refusal at 88.5-98.0% and HarmBench ASR at 1.0-2.5%; trigger lowers refusal (e.g., DeepSeek 96.5% to 56.5%).
• Ablation on DeepSeek safety: LCDD + circuit trigger yields -40pp refusal reversal vs -21pp for plain SFT with the same trigger; mask-only reaches only 28-34% sparsity vs 81% for joint optimization.

Surprising findings

• The same trigger optimization reverses behavior less effectively on plain SFT models than on LCDD carriers, and mask-only variants reach only about a third of the joint-optimization sparsity.
• Behavioral reversal and distributional reversion are partially independent: Vicuna reaches KL(Base||Trig)=0.014 with only moderate refusal suppression.

Limitations

• Trigger is a continuous soft prompt rather than discrete hard tokens.
• Structural necessity ablation performed on a single model and task combination.
• Method could be used to remove safety alignment from deployed models (dual-use risk).

Experimental setup

Datasets: Alpaca WildJailbreak Shakespearean and Modern English Conversational Dataset HarmBench

Benchmarks: HarmBench WildGuard MMLU HellaSwag Shakespeare LLM-judge

Metrics: fixed-response rate WildGuard refusal rate WildGuard harmfulness rate HarmBench ASR LLM-judge style score MMLU HellaSwag KL divergence

Baselines: Mask-only (CE) Mask-only (KL) output-only trigger plain SFT trigger

Builds on: task arithmetic SafeSeek straight-through estimator WildJailbreak

Closest papers by meaning

• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.88)
• Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.87)
• From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning (ICML 2024 · similarity 0.86)
• Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior (NeurIPS 2026 · similarity 0.86)
• Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis (ICML 2025 · similarity 0.86)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.86)