Research map / Large language models / Instruction and fine-tuning

On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes on-policy consistency training (OPCT), which trains a student on its own sampled responses to perturbed prompts using per-token reverse KL to a teacher seeing the clean prompt, and reports better generalization and less capability regression than SFT-based consistency training on sycophancy, jailbreaking, and safety awareness.

Problem

Offline, SFT-based consistency training memorizes surface forms of its training distribution, generalizes poorly to novel inputs and regresses general capabilities when targeting sycophancy, jailbreaks, and safety-fact elicitation.

Approach

Compute the consistency objective over the student's own on-policy rollouts on the perturbed prompt, scored per token by the same model conditioned on the clean contrastive prompt, using per-token reverse KL; no offline teacher demonstrations are needed.

Key results

• On sycophancy, OPCT reduces sycophantic rate to 8.1% vs 15.4% baseline and 11.2% for SFT (as stated in abstract).
• Under an adaptive per-target GRPO attacker, OPCT keeps jailbreak defense near 99% on held-out behaviors versus about 87% average for SFT/BCT-style baseline.
• SFT induces a 28-point drop on MATH-500 in cases where OPCT largely avoids such regressions.

Surprising findings

• On gpt-oss-20b, BCT collapses capability to single-digit accuracy on GPQA-Diamond, IFEval, and MATH-500 while OPCT preserves substantial capability.
• Llama-3.1-8B Safe-SAGE-Eval AUSC drops below baseline under OPCT (0.89 to 0.81), indicating some overrefusal.

Limitations

• Teacher must reliably elicit desired behavior under contrastive prompt; OPCT cannot add knowledge beyond the teacher's behavior.
• Capability evaluation is narrow (GPQA-Diamond, IFEval, MATH-500) and excludes code and long-context tasks.
• Higher compute overhead per step than SFT due to rollouts and teacher passes; only compared against consistency-training baselines, not RLHF/DPO/RL.

Experimental setup

Datasets: MMLU-derived sycophancy dataset HarmBench PAIR jailbreak datasets SAGE-Eval GPQA-Diamond IFEval MATH-500

Benchmarks: MMLU-derived sycophancy test set HarmBench static and adaptive jailbreak test SAGE-Eval Safe-SAGE-Eval GPQA-Diamond IFEval MATH-500

Metrics: sycophantic rate biased reasoning rate invariance rate defense win rate (1-ASR) AUSC accuracy

Baselines: BCT SFT Baseline (no training) ACT

Builds on: bias-augmented consistency training (BCT) on-policy distillation on-policy self-distillation (OPSD) activation consistency training (ACT)

Closest papers by meaning

• Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.89)
• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.88)
• Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications (ICML 2024 · similarity 0.88)
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.88)
• STAIR: Improving Safety Alignment with Introspective Reasoning (ICML 2025 · similarity 0.88)
• Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMs (NeurIPS 2026 · similarity 0.88)