Research map / Large language models / Language, safety and interpretability
SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SG-Bench, a safety benchmark that jointly tests LLM safety on generative and discriminative tasks (open-ended generation, multiple-choice, safety judgment) under jailbreak and prompt-engineering variants, finding poor cross-task and cross-prompt safety generalization.
Problem
Existing LLM safety benchmarks evaluate only generative or only discriminative safety and ignore how common prompting techniques (system prompts, few-shot demonstrations, chain-of-thought) affect safety, so safety generalization across tasks and prompt types is unmeasured.
Approach
Combine generation and discrimination evaluation subsets built from the same 1,442 malicious queries, and extend them with jailbreak templates and prompt-engineering variants (role/task-oriented system prompts, few-shot, CoT) to measure safety generalization; analyze failures with attention-by-gradient and embedding-distance analyses.
Key results
- Claude-3-haiku had the lowest average failure rate (2.99%), while Qwen1.5-7B-Chat had the highest (30.79%).
- Most models performed worse on discrimination tasks than on original-query generation; e.g., LLAMA3-8B-Instruct had 66.08% failure on multiple-choice vs 2.08% on original queries.
- Chain-of-thought prompting raised Claude-3's multiple-choice failure rate from 4.30% to 13.87%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Switch from original queries to discrimination tasks (multiple-choice and safety judgments) | safety-aligned LLMs · SG-Bench | Failure Rate | Hurt |
| Add role-oriented prompt (ablation) | LLMs · Jailbreak Attack | Failure Rate | Helped |
| Add task-oriented prompt (ablation) | LLMs · Jailbreak Attack | Failure Rate | Mixed |
| Add few-shot demonstrations (ablation) | LLMs · Jailbreak Attack | Failure Rate | Hurt |
| Add few-shot demonstrations (ablation) | LLMs · Multiple-choice Questions and Safety judgments | Failure Rate | Helped |
| Add chain-of-thought prompting (ablation) | claude-3-haiku-20240307 · Multiple-choice Questions | Failure Rate: 4.30 -> 13.87 | Hurt |
| Add chain-of-thought prompting (ablation) | InternLM2-7B-Chat · Safety judgments | Failure Rate: +17.33 | Hurt |
| Increase model size from 7B to 13B | LLAMA2-Chat · SG-Bench average | Avg FR: 23.73 -> 18.21 | Helped |
| Use AutoDAN instead of SG-Bench-6 jailbreaks (ablation) | LLAMA2-7B-Chat and Qwen1.5-7B-Chat · Jailbreak Attack | Failure Rate: 11.75 -> 33.29; 39.35 -> 81.38 | Hurt |
| Use GCG instead of SG-Bench-6 jailbreaks (ablation) | LLAMA2-7B-Chat and Qwen1.5-7B-Chat · Jailbreak Attack | Failure Rate: 11.75 -> 2.67; 39.35 -> 10.11 | Helped |
Surprising findings
- Safety-aligned LLMs that refuse nearly all original harmful queries still fail often on discriminative safety judgments and multiple-choice tests.
- Chain-of-thought prompting generally harmed safety discrimination performance.
Limitations
- Relies on LlamaGuard-7B as an imperfect automated judge for generation outputs; evaluator choice changes absolute scores.
- Coarse six-category safety taxonomy without analysis of fine-grained safety issue effects.
- Jailbreak coverage limited to six templates in the main benchmark; advanced attacks only supplementary.
Experimental setup
Datasets: AdvBench HarmfulQA Beaver-eval SaladBench alpaca
Benchmarks: SG-Bench
Metrics: failure rate average failure rate
Baselines: SaladBench SafetyBench DecodingTrust EasyJailbreak Jailbroken
Builds on: AdvBench HarmfulQA Beaver-eval SaladBench PromptBench
Closest papers by meaning
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.91)
- Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets (ICML 2025 · similarity 0.91)
- Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms (NeurIPS 2025 · similarity 0.90)
- Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? (ICLR 2025 · similarity 0.90)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.90)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.