Research map / Large language models / Language, safety and interpretability
Safety alignment of LLMs
Study how safety training, safeguards, and unsafe-data forgetting shape LLM behavior.
123 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in Language, safety and interpretability, keyword cluster “representations · interpretability · heads”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 107 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 2.3% to 5.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · 25 citations)
- 2025 · Security Attacks on LLM-based Code Completion Tools (AAAI 2025 · 12 citations)
- 2026 · MPMA: Preference Manipulation Attack Against Model Context Protocol (AAAI 2026 · 6 citations)
Earliest papers here
- WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · 25 citations)
- LLMGuard: Guarding against Unsafe LLM Behavior (AAAI 2024 · 14 citations)
- SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (NeurIPS 2024 · 13 citations)
What has been tried
56 papers here report 608 measured changes (a component added, removed or swapped, compared with a baseline): helped 60%, no clear effect 8%, hurt 29%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Remove synthetic adversarial data | WildGuardTest prompt harmfulness (Adversarial) · F1 (%) | 85.5 -> 77.1 | Hurt | WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (NeurIPS 2024 · 25 citations) |
| Add role-oriented prompt | Jailbreak Attack · Failure Rate | Helped | SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (NeurIPS 2024 · 13 citations) | |
| Add guiding words to the Level 1 attack | Level 1 jailbreaking · ASR | 7.50% (-91.9%) | Helped | Security Attacks on LLM-based Code Completion Tools (AAAI 2025 · 12 citations) |
| Instruct the model to be helpful and safe using a system prompt | OR-Bench-Toxic · toxic prompt rejection rate | around 35% more toxic prompts | Helped | OR-Bench: An Over-Refusal Benchmark for Large Language Models (ICML 2025 · 9 citations) |
| Change the message-role assignment | Prompt hijacking and extraction benchmarks · Performance across metrics | No clear effect | Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game (ICLR 2024 · 8 citations) | |
| Apply chat alignment | MLLMGuard · ASD | Helped | MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · 7 citations) | |
| Fine-tune GPT-3.5-turbo on human judge annotations | SORRY-Bench human-judgment test set · Cohen Kappa agreement | 53.4 -> 83.8 | Helped | SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · 6 citations) |
| Remove the refusal objective L_r | AdvBench · % compliance | 1.4 -> 10.0 | Hurt | On Prompt-Driven Safeguarding for Large Language Models (ICML 2024 · 6 citations) |
| steer middle rather than latter layers | XSTest · Safe refusal rate | 12.00 -> 9.20 | Helped | SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering (AAAI 2025 · 5 citations) |
| increase the number of attacked tokens from 1 to 20 | Harry Potter Q&A · C-ASR | 30.9 -> 25.5; 25.5 -> 20 | Hurt | Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · 5 citations) |
| use DPO instead of SFT for the Target update | SEAS-Test, Beaver, and HH-RHLF · ASR (%) | 67.90 -> 21.35; 25.43 -> 7.14; 20.86 -> 3.80 | Helped | SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Models (AAAI 2025 · 4 citations) |
| Use a newer Llama model version | RTP-LX toxic prompts · agreement with human annotators | Helped | RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios? (AAAI 2025 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 57 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- AdvBench · 16 of 57 papers
- XSTest · 13 of 57 papers
- HarmBench · 13 of 57 papers
- BeaverTails · 9 of 57 papers
- Alpaca · 8 of 57 papers
- JailbreakBench · 8 of 57 papers
- MMLU · 7 of 57 papers
- GSM8K · 6 of 57 papers
Baselines
- GPT-4o · 8 of 57 papers
- Llama-Guard-3-8B · 7 of 57 papers
- GCG · 5 of 57 papers
- Llama Guard · 4 of 57 papers
- LAT · 4 of 57 papers
- PAIR · 4 of 57 papers
- Self-CD · 3 of 57 papers
- SafeDecoding · 3 of 57 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 123 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 6 papers
- Tsinghua University · 6 papers
- University of California, Berkeley · 5 papers
- Google · 5 papers
- Harbin Institute of Technology · 5 papers
- National University of Singapore · 4 papers
- University of Illinois, Urbana Champaign · 4 papers
- University of California, Los Angeles · 4 papers
- Meta · 4 papers
- Zhejiang University · 4 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Mechanistic analysis of attention and representations (43 papers)
- Jailbreak attacks and defenses (57 papers)
Elsewhere in the topic
- LLM unlearning and benchmarks (298 papers)
- LLM pruning and scaling data (285 papers)
- Controlled text generation (220 papers)
- Faithfulness of LLM explanations (169 papers)
- Data and evaluation methodology (136 papers)
- Uncertainty quantification for LLMs (136 papers)
- Pruning and circuits in LLMs (135 papers)
- Statistical frameworks for LLM evaluation (124 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).