Research map / Large language models / Language, safety and interpretability
Backtracking Improves Generation Safety
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes backtracking, where a language model emits a special [RESET] token after an unsafe partial generation and regenerates a safe response, trained via SFT and DPO on safety and utility data.
Problem
Safety alignment is prevention-based, so once a language model begins an unsafe generation it tends to continue it; the paper asks how to improve safety when unsafe generations are expected to occur.
Approach
Teach the model to emit a one-shot [RESET] token after recognizing an unsafe partial output, discard everything before the reset, and regenerate a safe response, learned via SFT on contrastive demonstrations and DPO on backtracking preference pairs.
Key results
- Backtracking Llama-3-8B with SFT+DPO reduces overall safety violations from 5.3% to 1.5% on the 1,033-prompt safety set, with MT-Bench 7.12 vs 6.68 for baseline.
- For Gemma-2-2B, backtracking SFT+DPO lowers overall violations from 10.8% to 6.1%.
- Against prefilling, backtracking reduces attack success on Llama from 85.0% to 2.6%, while GCG and AutoDAN success rates also drop, e.g., GCG on Llama 62.5% to 16.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add backtracking to SFT + DPO | Gemma-2-2B · Safety evaluation, Overall | violation rate: 10.8% -> 6.1% | Helped |
| Add backtracking to SFT + DPO | Llama-3-8B · Safety evaluation, Overall | violation rate: 5.3% -> 1.5% | Helped |
| Add backtracking to SFT + DPO | Gemma-2-2B · MT-Bench | score: 5.20 -> 4.96 | No clear effect |
| Add backtracking to SFT + DPO | Llama-3-8B · MT-Bench | score: 6.68 -> 7.12 | No clear effect |
| Use backtracking SFT without DPO (ablation) | Gemma-2-2B · Safety evaluation, Overall | violation rate: 10.6% -> 9.0% | Helped |
| Use backtracking SFT without DPO (ablation) | Llama-3-8B · Safety evaluation, Overall | violation rate: 5.3% -> 4.8% | Helped |
| Add DPO after backtracking SFT (ablation) | Gemma-2-2B · Safety evaluation, Overall | violation rate: 9.0% -> 6.1% | Helped |
| Add DPO after backtracking SFT (ablation) | Llama-3-8B · Safety evaluation, Overall | violation rate: 4.8% -> 1.5% | Helped |
| Apply a more negative [RESET] logit bias (ablation) | Backtracking Llama-3-8B · Safety evaluation | safety: 2X as unsafe | Hurt |
| Apply a small [RESET] logit bias (-5) (ablation) | Backtracking Llama-3-8B · Safety evaluation | generation efficiency: -5 | Helped |
| Train backtracking with a 50:50 utility/safety data mixture (ablation) | Backtracking model · Safety tuning | helpfulness | Hurt |
| Use backtracking rather than baseline DPO | Gemma-2-2B · Prefilling attack | attack success rate: 50.4% -> 11.6% | Helped |
Surprising findings
- Backtracking SFT alone is only marginally effective; the gains come mainly from DPO, which raises reset rates to about 50-60% on safety prompts.
- AutoDAN is a stronger attack against backtracking models than the paper's own adaptive attack.
Limitations
- Even with backtracking, models remain far from adequate adversarial robustness; optimization-based attacks reduce its effectiveness.
- Backtracking incurs latency and throughput cost due to discarded tokens and false-positive resets.
- Evaluated only on two base models and safety-focused benchmarks; relies on a safety classifier for training labels.
Experimental setup
Datasets: OpenAssistant-2 HH-RLHF harmless subset AdvBench MaliciousInstructions SimpleSafetyTests StrongREJECT MT-Bench
Benchmarks: AdvBench MaliciousInstructions SimpleSafetyTests StrongREJECT MT-Bench
Metrics: safety violation rate attack success rate reset rate MT-Bench score latency throughput
Baselines: Baseline SFT Baseline SFT + DPO Backtrack SFT GCG AutoDAN Prefilling
Builds on: DPO GCG AutoDAN Llama Guard 2
Closest papers by meaning
- Reinforcement Learning with Backtracking Feedback (NeurIPS 2025 · similarity 0.90)
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.90)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.90)
- Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories (NeurIPS 2026 · similarity 0.89)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.89)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.