Research map / Large language models / Language, safety and interpretability
SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests
Punya Syon Pandey, Lê Hải Sơn, Devansh Bhardwaj, Zhijing Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate on sociopolitical prompts instead of generic harm | open-weight LLMs · SocialHarmBench | HarmBench ASR: Mistral-7B 62.50 on historical revisionism | Hurt |
| compare closed vs open-weight models | Claude-Sonnet-4 vs Mistral-7B-Instruct · SocialHarmBench combined | HarmBench ASR: 27.71 -> 0.78 | Improved |
| apply weight-tampering attack | open-weight LLMs · SocialHarmBench | HarmBench score: above 0.90 across nearly all models | Hurt |
| apply latent-space and input-space attacks (AutoDAN, GCG, SoftOpt, LAT) | open-weight LLMs · SocialHarmBench | HarmBench score | Mixed |
| prompts set in the 21st century | LLMs · SocialHarmBench | HB / SR: HB=0.67, SR=0.50 | Hurt |
| prompts tied to Latin America, USA, UK | Mistral, Gemma, DeepSeek · SocialHarmBench | HB: HB 0.50-0.75 | Hurt |
Closest papers by meaning
- SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.86)
- ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety (NeurIPS 2026 · similarity 0.86)
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.85)
- On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.85)
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.85)
- Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
