Research map / Large language models / Language, safety and interpretability
Safety Depth in Large Language Models: A Markov Chain Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames autoregressive LLM safety alignment as a Markov chain with refusal states made absorbing via iterative biased transition updates, extends to cyclic group data augmentation and ensembles, and validates with toy chains and small open LLMs judged by Llama 3.2 1B.
Problem
Determining how deep safety alignment should extend (beyond initial output tokens) so harmful content probability is bounded, given that shallow alignment is fragile to jailbreaks and benign fine-tuning.
Approach
Model autoregressive LLM fine-tuning as iterative updates to a Markov transition matrix Q_t = Q_0 + alpha*gamma^t*B, derive conditions under which refusal states become delta-absorbing, extend with cyclic-permutation data augmentation and ensembles with union/average/majority aggregation.
Key results
- Gemma 2B safety score mean rises from 0.35 (not aligned) to 0.42 (shallow), 0.46 (deep), and 0.61 (cyclic) on HEx-PHI.
- Toy 4-state Markov chain: escape probability decreases from about 0.75 to below 0.01 over 50 bias applications.
- For Qwen 2.5, union, average, and majority ensembles of three shallow models score higher, clustering around 0.9-1.0, than the single deep model with lower median.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply shallow safety alignment instead of no safety alignment | Gemma 2B · HEx-PHI | safety score: 0.35 ± 0.05 -> 0.42 ± 0.06 | Helped |
| Use deep instead of shallow safety alignment | Gemma 2B · HEx-PHI | safety score: 0.42 ± 0.06 -> 0.46 ± 0.05 | Helped |
| Use cyclic group augmentation instead of deep safety alignment | Gemma 2B · HEx-PHI | safety score: 0.46 ± 0.05 -> 0.61 ± 0.06 | Helped |
| Use cyclic group augmentation instead of shallow safety alignment | Gemma 2B · HEx-PHI | safety score: 0.42 ± 0.06 -> 0.61 ± 0.06 | Helped |
Surprising findings
- Cyclic group augmentation of permutation-based bias can make non-refusal states absorbing, potentially hurting utility, per the authors' remark.
Limitations
- Relatively small models and datasets leave scalability to large models open.
- Real-world utility was not exhaustively tested.
- Theoretical assumptions (fine-tuning equals iterative transition matrix update, absorbing refusal) are idealized.
Experimental setup
Datasets: MaliciousInstruct HEx-PHI
Benchmarks: HEx-PHI
Metrics: safety score
Baselines: shallow alignment deep alignment
Builds on: shallow safety alignment data augmentation Markov chain view of autoregressive LLMs neural tangent kernel
Closest papers by meaning
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.93)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.92)
- Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.91)
- Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories (NeurIPS 2026 · similarity 0.91)
- Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (ICLR 2025 · similarity 0.90)
- UpSafe℃: Upcycling for Controllable Safety in Large Language Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.