Research map / Large language models / AI and society
PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harm
Jing-Jing Li, Joel Mire, Eve Fleisig, Valentina Pyatkin, Anne Collins, Maarten Sap, Sydney Levine
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| personalize k-shot steering to individual annotators instead of aggregated ratings | GPT-4.1 k-shot · PluriHarms | MAE: 0.254 -> 0.196 | Improved |
| use k-shot examples instead of value profile summaries for individual steering | GPT-4.1 · PluriHarms (individual) | MAE: 0.233 -> 0.196 | Improved |
| steer GPT-4.1 to aggregated ratings with value profile instead of zero-shot | GPT-4.1 · PluriHarms (aggregated) | MAE: 0.263 -> 0.260 | No clear effect |
| align SafetyAnalyst to aggregated ratings rather than WildGuard probability zero-shot | SafetyAnalyst · PluriHarms (aggregated) | MAE: 0.364 -> 0.361 | No clear effect |
| use WildGuard probabilities instead of binary classification labels | WildGuard-7B · PluriHarms (aggregated) | MAE: 0.403 -> 0.364 | Improved |
| use general model (GPT-4.1 zero-shot) instead of specialized safety model SafetyAnalyst | safety models · PluriHarms (aggregated) | MAE: 0.361 -> 0.263 | Improved |
| personalize SafetyAnalyst to individual annotators | SafetyAnalyst-8B · PluriHarms | MAE: 0.361 -> 0.311 | Improved |
| add prompt-level features to the null model (ablation) | mixed-effects regression · PluriHarms | BIC | Improved |
| add annotator traits and trait interactions on top of prompt features (ablation) | mixed-effects regression · PluriHarms | BIC | Improved |
| add annotator traits to explain ratings (ablation) | mixed-effects model (random intercept per prompt) · PluriHarms | R2 (fixed effects): 0.0232 -> 0.0686 | Improved |
| increase synthetic harm level of prompts | DeepSeek-V3 generated prompts · PluriHarms | Spearman correlation with human rating: r = 0.59 | Improved |
Closest papers by meaning
- Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models (NeurIPS 2025 · similarity 0.87)
- SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior (ICML 2025 · similarity 0.87)
- OpenDeception: Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation (ICML 2026 · similarity 0.87)
- Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties (AAAI 2024 · similarity 0.86)
- BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments (NeurIPS 2026 · similarity 0.86)
- Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
