Research map / Large language models / AI and society
Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DIVE, a 1000-prompt-image text-to-image safety dataset rated by 637 demographically intersectional raters, showing demographic differences in harm perception and testing Gemma-based LLM-raters steered with demographic prompts.
Problem
Safety evaluations of text-to-image models often rely on policy raters or single annotators, neglecting diverse rater backgrounds and the subjective nature of harm, so models may be misaligned with significant user populations.
Approach
Build a pluralistic T2I safety dataset with uniform recruitment across 30 gender-age-ethnicity trisections, multi-format feedback (Likert harm to self and others, violation type, comments), and analyze demographic effects, classifier blind spots, and demographic-prompted LLM raters.
Key results
- Overall inter-rater reliability across 637 raters was 0.24, indicating high disagreement on harm perception.
- Black-GenZ and Black-Millennial intersectional groups had the highest GAI values (1.38 and 1.29), while broad GenZ had 1.03.
- Demographic prompting raised Kendall's Tau correlation of the LLM-rater with human ratings from about 0.02 (no demographics) to about 0.23 on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Include demographic information in the prompt (ablation) | Gemma 4B LLM-rater · DIVE human-rating comparison | Kendall's Tau: 0.024 -> 0.23 | Helped |
| Add in-context examples to demographic prompting (ablation) | Gemma 4B LLM-rater · DIVE human-rating comparison | Kendall's Tau: .222 -> .222; .243 -> .242; .216 -> .225; .229 -> .226; .252 -> .245; .217 -> .209; .225 -> .210; .247 -> .252; .240 -> | Mixed |
| Use Women rather than Men as the rater group | Human T2I safety raters · DIVE, harm to self | probability of assigning a higher harm score: 0.55 | Helped |
| Use Black rather than White raters | Human T2I safety raters · DIVE, harm to self | probability of assigning a higher harm score: 0.57 | Helped |
| Use Women rather than Men as the rater group | Human T2I safety raters · DIVE, harm to others | probability of assigning a higher harm score: 0.52 | Helped |
| Use Black rather than White raters | Human T2I safety raters · DIVE, harm to others | probability of assigning a higher harm score: 0.53 | Helped |
| Compare intersectional demographic groups rather than broad single-demographic groups | Human T2I safety raters · DIVE, group cohesion | Group Association Index: 1.03; 1.38 | Helped |
| Use Women rather than Men as the rater group | Human T2I safety raters · DIVE, safety-violation simulation | unique unsafe cases across violation types: 9-14 pairs | Helped |
| Use SouthAsian, Latinx, or Black raters rather than other rater groups | Human T2I safety raters · DIVE, Bias | unique unsafe cases: 24-28 pairs | Helped |
| Use White raters rather than non-White raters | Human T2I safety raters · DIVE, Bias | cases missed that non-White raters flagged: 17 Bias cases | Hurt |
| Evaluate with diverse raters rather than conventional safety classifiers | T2I safety evaluation · DIVE, Bias | false negative rate | Helped |
| Use demographic rather than non-demographic prompting (ablation) | Gemma 4B LLM-rater · DIVE human-rating comparison | Kendall's Tau: 0.024 -> 0.23 | Mixed |
Surprising findings
- Safety classifiers and policy raters show high false negative rates on bias harms relative to diverse raters.
- Gender-age intersections showed no statistically significant group cohesion, suggesting ethnicity-based intersections matter more.
Limitations
- Only gender, age, and ethnicity are used as rater representations; other value or cultural factors are not captured.
- Only small open-source LLMs (Gemma 4B) were tested, with low overall human-simulation accuracy.
- Policy raters' demographics were not collected, so expertise and lived experience cannot be disentangled.
Experimental setup
Datasets: Adversarial Nibbler
Benchmarks: DIVE
Metrics: Likert harmfulness scores inter-rater reliability (IRR) cross-replication reliability (XRR) Group Association Index (GAI) false negative rate Kendall's Tau (b)
Baselines: LlavaGuard ShieldGemma policy raters
Builds on: Adversarial Nibbler DICES
Closest papers by meaning
- VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.88)
- SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset (NeurIPS 2024 · similarity 0.87)
- Position: A Roadmap to Pluralistic Alignment (ICML 2024 · similarity 0.87)
- ELITE: Enhanced Language-Image Toxicity Evaluation for Safety (ICML 2025 · similarity 0.87)
- SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.87)
- Multimodal Situational Safety (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.