Research map / Large language models / Language, safety and interpretability
Safety Pretraining: Toward the Next Generation of Safe AI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a data-centric safety pretraining framework (safety filtering, synthetic rephrasing, refusal and moral-education data, and harmfulness-tag training with safe beam search) that yields 1.7B SafeLM models with lower attack success rates and robustness to benign finetuning.
Problem
LLMs deployed in high-stakes settings generate harmful content, and post-hoc alignment is brittle because unsafe patterns learned during pretraining are hard to remove.
Approach
Embed safety into pretraining data and training itself: classify and score web data, rephrase unsafe text into educational context, add refusal and moral-education data, and insert a special harmfulness tag that is steered against at decoding time (Safe Beam Search).
Key results
- Safety-pretrained models reduce base-model ASR from 44% (standard pretraining) to 11%, and the abstract reports ASR falling from 38.8% to 8.4% after benign finetuning.
- Harmfulness-tag pretraining keeps ASR near 8% after benign GSM8K finetuning versus ~30% for tag-annotated instruction tuning only.
- Ablations show score-0-only training raises ASR to 43.8%, while rephrasing, refusal, and moral education data lower it to 33.6%, 25.1%, and 20.0% respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train only on score-0 examples instead of raw data (ablation) | Safety pretraining · GSM8K after benign finetuning | ASR: 38.8% -> 43.8% | Hurt |
| Add rephrased score 1–3 data (ablation) | Safety pretraining · GSM8K after benign finetuning | ASR: 38.8% -> 33.6% | Helped |
| Add refusal completions for score-4 and score-5 data (ablation) | Safety pretraining with rephrased data · GSM8K after benign finetuning | ASR: 33.6% -> 25.1% | Helped |
| Add moral education data (ablation) | Safety pretraining with rephrased and refusal data · GSM8K after benign finetuning | ASR: 25.1% -> 20.0% | Helped |
| Annotate unsafe content with a harmfulness tag during pretraining rather than during IFT (ablation) | 1.7B model · GSM8K after benign finetuning | ASR: ~30% -> ~8% | Helped |
| Use harmfulness-tag injection at 5% of sequence positions rather than the other tested rates (ablation) | Safety pretraining with data-centric interventions · GSM8K after benign finetuning | ASR | Helped |
| Safety-pretrain rather than use standard pretraining | 1.7B base model · Base Model Safety Evaluations | ASR: 44% -> 11% | Helped |
| Add rephrasing, refusal, and moral education data | 1.7B model · Standard benchmarks | Avg.: 42.8% -> 42.9% | No clear effect |
Surprising findings
- Training only on the safest subset (score 0) increased attack success rate from 38.8% to 43.8% relative to raw data.
- Adding safety instruction tuning gives near-zero ASR that rises sharply after a small amount of benign finetuning for standard models.
Limitations
- SafeBeam is evaluated only against jailbreaks not adapted to the inference algorithm, and slightly increases overrefusal.
- Safety classifiers over-score benign content and LLM-based scorers over-flag, causing false positives.
- Results are reported for a 1.7B model on a limited set of benchmarks.
Experimental setup
Datasets: FineWeb-Edu StackOverflow FineMath Cosmopedia Ultrachat-200k WildGuardMix WildJailbreak toxigen-data Jigsaw Toxic Comments GSM8K Alpaca
Benchmarks: HarmBench TDC JailbreakBench AdvBench Alpaca arc_challenge arc_easy commonsense_qa gsm8k openbookqa piqa triviaqa
Metrics: attack success rate recall F1 overrefusal rate accuracy
Baselines: LLaMA Guard Profanity Checker SmolLM-1.7B-Instruct Raw-data pretraining
Builds on: SmolLM2 FineWeb-Edu Constitutional AI DPO LitGPT
Closest papers by meaning
- RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.91)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.91)
- On Prompt-Driven Safeguarding for Large Language Models (ICML 2024 · similarity 0.91)
- Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (ICLR 2025 · similarity 0.90)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
- HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.