Research map / Large language models / Language, safety and interpretability
SAGE-Eval: Evaluating LLMs for Systematic Generalizations of Safety Facts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SAGE-Eval, a benchmark of 104 safety facts expanded into 10,428 naive user scenarios, showing frontier LLMs fail to consistently apply well-established safety facts (best model passes 57.69%).
Problem
Whether LLMs robustly generalize well-established safety facts to novel naive user queries where the risk is implicit rather than explicitly harmful.
Approach
A benchmark of safety facts from reputable organizations, converted via GPT-4o into naive user questions with hidden risks, validated by human annotators, and augmented with tones and typos; uses an LLM-as-judge (o3-mini with Gemini re-check) and a model-level all-variants-must-pass metric plus area under safety curve, and power-law forecasting of failure rates at larger prompt counts.
Key results
- Claude-3.7-Sonnet achieves the top model-level safety score of 57.69%, while o1 scores 41.35% and o3-mini 34.62%.
- Model capability (Chatbot Arena R^2=0.218) and training compute (FLOPS R^2=0.220) only weakly correlate with SAGE-Eval scores.
- Bias-corrected power-law forecasting predicts the strongest model achieves only about 38% model-level safety score at n=1,000 prompts per fact.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a reflection system prompt (ablation) | Claude-3.7-Sonnet on SAGE-Eval · SAGE-Eval | model-level safety score: 57.69 (±4.84) -> 77.88 (±4.07); 20.19 | Helped |
| add a reflection system prompt (ablation) | Gemini-1.5-Pro on SAGE-Eval · SAGE-Eval | model-level safety score: 47.12 (±4.89) -> 48.08 (±4.08); 0.96 | No clear effect |
| add a reflection system prompt (ablation) | GPT-4o on SAGE-Eval · SAGE-Eval | model-level safety score: 24.76 (±4.21) -> 36.54 (±4.72); 11.78 | Helped |
| fine-tune OLMo-2-32B-SFT with DPO (ablation) | OLMo-2-32B on SAGE-Eval · SAGE-Eval | Area under Safety Curve: 0.132 -> 0.226 | Helped |
| use a 100-word context with hidden safety concerns instead of instruction-only prompts (ablation) | models evaluated on SAGE-Eval · SAGE-Eval prompt types | average fact-level safety score: 0.896 -> 0.803 | Hurt |
| use a 50-word context with hidden safety concerns instead of instruction-only prompts (ablation) | models evaluated on SAGE-Eval · SAGE-Eval prompt types | average fact-level safety score: 0.896 -> 0.825 | Hurt |
| use a short unsafe context instead of instruction-only prompts (ablation) | models evaluated on SAGE-Eval · SAGE-Eval prompt types | average fact-level safety score: 0.896 -> 0.831 | Hurt |
| add a depressed tone to prompts (ablation) | models evaluated on SAGE-Eval · SAGE-Eval augmentation types | average fact-level safety score: 0.907 -> 0.865 | Hurt |
| add a happy tone to prompts (ablation) | models evaluated on SAGE-Eval · SAGE-Eval augmentation types | average fact-level safety score: 0.907 -> 0.886 | Hurt |
| add typos to prompts (ablation) | models evaluated on SAGE-Eval · SAGE-Eval augmentation types | average fact-level safety score: 0.907 -> 0.900 | Hurt |
| use power-law forecasting instead of logistic or exponential forecasting (ablation) | SAGE-Eval safety-score forecasting · SAGE-Eval forecasting at n=100 | average absolute difference: 14.82 vs. 34.47 vs. 53.50 | Helped |
| apply model-specific bias correction to power-law predictions (ablation) | SAGE-Eval safety-score forecasting · SAGE-Eval forecast test set | average absolute percentage error: 32.97 -> 5.86 | Helped |
Surprising findings
- Safety scores do not correlate significantly with training compute or Chatbot Arena capability, and fact frequency in pre-training data (WIMBD, Google search) shows no significant correlation with fact-level safety.
- A reflection system prompt helps Claude-3.7-Sonnet substantially (+20.19 points) but barely helps Gemini-1.5-Pro (+0.96).
Limitations
- Root-cause analyses (fact frequency) were inconclusive, suggesting other factors matter.
- Benchmark construction relied partly on LLMs (GPT-4o), potentially introducing bias and not covering all real-world query complexities.
- Reflection system prompt showed limited benefit and risks making models overly vigilant.
Experimental setup
Datasets: SAGE-Eval
Benchmarks: SAGE-Eval
Metrics: model-level safety score area under safety curve fact-level safety score
Baselines: o1 o3-mini o1-mini gpt-4o gpt-4o-mini gpt-3.5-turbo claude-3-5-sonnet claude-3-7-sonnet gemini-1.5-pro DeepSeek-V3
Builds on: HarmBench AdvBench WMDP Chatbot Arena
Closest papers by meaning
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.86)
- CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models (ICML 2025 · similarity 0.86)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.85)
- SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (NeurIPS 2024 · similarity 0.85)
- BingoGuard: LLM Content Moderation Tools with Risk Levels (ICLR 2025 · similarity 0.85)
- Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.