Research map / Large language models / Language, safety and interpretability

Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that short token injections at any decoding step make safety-aligned LLMs produce harmful output, and proposes bidirectional trajectory augmentation with iterative SimPO (QLoRA) training that reduces injection ASR toward zero and transfers to prefilling, suffix, and PAIR attacks.

Problem

Safety-aligned LLMs can be redirected toward harmful outputs by short token injections at arbitrary mid-generation steps, and existing alignment methods do not train on the dynamics of disrupted generation; refusal-like hidden states do not guarantee robustness.

Approach

Generalize shallow safety to injection at any decoding step, then construct training trajectories by bidirectionally injecting harmful and refusal phrases at points where the hidden state enters refusal or accept regions, and train with reference-free SimPO so the model learns to resist harmful redirection and recover to safe generation.

Key results

• Single harmful injection raises Llama-3.1-8B ASR on AdvBench from 0.00 to 92.12 (OM) and Qwen from 1.54 to 88.08; trajectory alignment reduces injection ASR on Llama AdvBench to 4.42 (OM) / 0.19 (LG).
• On Qwen2.5-7B, prefilling ASR on AdvBench (LG) drops from 84.62 to 0.00 after trajectory alignment.
• Benign MMLU and PROST performance of aligned models stays within 1.1 points of baselines.

Surprising findings

• Refusal-prototype similarity before injection does not predict robustness; for Mistral the highest ASR coincides with the highest refusal similarity.
• Benign phrases such as 'Certainly!' raise base Llama ASR to 51.54, showing displacement itself is a vulnerability.

Limitations

• Mistral prefilling ASR under OpenAI moderation remains high after alignment (77.69 on AdvBench), so robustness is incomplete.
• Circuit Breakers and LAT are evaluated only with released checkpoints on a subset of models, and Egida-DPO lacks a Mistral checkpoint, making comparisons uneven.

Experimental setup

Datasets: AdvBench HarmBench HEx-PHI JailbreakBench MMLU PROST XSTest

Benchmarks: AdvBench HarmBench HEx-PHI JailbreakBench MMLU PROST XSTest

Metrics: attack success rate MMLU accuracy over-refusal rate

Baselines: Egida-DPO Circuit Breakers SafeProbing LAT PAIR Prefilling I-GCG

Builds on: SimPO Circuit Breakers Latent adversarial training Egida-DPO shallow safety alignment

Closest papers by meaning

• Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.92)
• Safety Depth in Large Language Models: A Markov Chain Perspective (NeurIPS 2025 · similarity 0.91)
• Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.90)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.90)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
• Backtracking Improves Generation Safety (ICLR 2025 · similarity 0.89)