Research map / Large language models / Language, safety and interpretability
Hair-Trigger Alignment: Black-Box Evaluation Cannot Guarantee Post-Update Alignment
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Eleni Triantafillou, Peter Kairouz, Salman Avestimehr, Sai Praneeth Karimireddy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adversarially train the model so a single benign gradient step reveals misaligned behavior | Llama3.2-3B-Instruct · AdvBench safety score | safety score: 0.954 (before) -> 0.085 (after update) | Hurt |
| Adversarially train the model so a single benign gradient step reveals misaligned behavior | Mistral-7B-Instruct-v0.2 · AdvBench safety score | safety score: 0.948 (before) -> 0.044 (after update) | Hurt |
| Adversarially train for post-update re-emergence of forgotten data | Llama3.2-3B-Instruct · TOFU forget set | leakage score: 0.000 -> 1.000 | Hurt |
| Adversarial post-update training for dishonesty | Llama3.2-3B-Instruct · TriviaQA | accuracy: 0.549 -> 0.062 | Hurt |
| Update with a disjoint Alpaca subset instead of the training update set (ablation) | fragile models · Jailbreak, honesty, privacy | alignment scores: Llama AdvBench 0.094; Mistral AdvBench 0.232 | No clear effect |
| Update with Dolly instead of Alpaca (ablation) | fragile models · Jailbreak, honesty, privacy | alignment scores: Llama TriviaQA 0.526; Mistral TOFU forget 0.175 | Mixed |
| Update with GSM8K instead of Alpaca (ablation) | fragile models · Jailbreak, honesty, privacy | alignment scores: Llama AdvBench 0.923; Mistral AdvBench 0.853 | Improved |
| Reduce step size from 1e-4 to 1e-5 for adversarial training and update (ablation) | fragile Llama3.2-3B · Jailbreak, honesty, privacy | alignment scores: AdvBench after update 0.09 -> 0.53 | Hurt |
| Increase LoRA rank from 2 to 16 (ablation) | fragile Llama3.2-3B · random sequences concealed then revealed | max concealed sequences | Improved |
| Benign update of original (non-adversarial) models | original models · All tasks | alignment scores: AdvBench 0.975 -> 0.973 (Llama) | No clear effect |
Closest papers by meaning
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.89)
- Alignment Dynamics in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.88)
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (ICLR 2025 · similarity 0.88)
- Perplexity-aware Correction for Robust Alignment with Noisy Preferences (NeurIPS 2024 · similarity 0.88)
- LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations (NeurIPS 2026 · similarity 0.87)
- Differentially Private Steering for Large Language Model Alignment (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
