Research map / Large language models / Language, safety and interpretability

HelpBench: Assessing the Ability of LLMs to Provide Privacy, Safety, and Security Advice

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds HelpBench, a 450-question benchmark of Reddit-derived privacy, safety, and security help questions with expert rubrics and an LLM auto-rater, and evaluates ten frontier LLMs.

Problem

Consumer-facing advice on digital privacy, safety, and security (PSS) is under-benchmarked, and flawed LLM advice in these high-stakes settings can cause account compromise, financial loss, or harm to at-risk users.

Approach

Rubric-based evaluation of PSS advice: each rephrased real-user question gets expert-written factual criteria with positive/negative point values, plus shared delivery criteria (importance placement, over-information, action overload, tone, overstated effectiveness), scored by an LLM auto-rater validated against experts.

Key results

• Across ten models, average rubric score is 81%; GPT 4.1 and GPT 5.0 score highest at 86%, while Qwen 3 scores lowest at 68%.
• Auto-rater achieves Pearson correlation 0.85 with expert scores (0.96 on factual criteria, 0.78 on delivery criteria) using Gemini 2.5 Pro.
• Models score highest on scams questions (0.87 average) and lowest on moderation tools (0.76 average); performance is higher on general educational questions than platform-specific ones.

Surprising findings

• Qwen 3 fails mainly on delivery criteria (83% negative delivery points lost), and Claude Sonnet models give incomplete answers despite few mistakes.
• Models sometimes escalate tone, suggest hyper-vigilance, or give dangerous advice in technology-facilitated abuse cases, such as suggesting a disguise to hide location from an abuser.

Limitations

• Possible training-data contamination since questions may have been seen by evaluated models.
• Rubrics become obsolete as platform UIs and features change; only one response per model evaluated.
• Auto-rater correlation is weaker on delivery criteria (0.78), and expert ratings may differ from non-expert preferences.

Experimental setup

Datasets: Reddit PSS post dataset (3M posts, 2021-2024, multi-label topic annotations)

Benchmarks: HelpBench

Metrics: rubric score (fraction of available points) Pearson correlation with human expert scores

Baselines: GPT 4.1 GPT 5.0 Claude Sonnet 4 Claude Sonnet 4.5 Gemini 2.5 Pro Gemini 3 Pro Grok 4 DeepSeek 3.1 GLM 4.6 Qwen 3

Builds on: WildBench HealthBench

Closest papers by meaning

• Empowering Users in Digital Privacy Management through Interactive LLM-Based Agents (ICLR 2025 · similarity 0.88)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.86)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · similarity 0.85)
• PrivAuditor: Benchmarking Data Protection Vulnerabilities in LLM Adaptation Techniques (NeurIPS 2024 · similarity 0.85)
• Combating Phone Scams with LLM-based Detection: Where Do We Stand? (Student Abstract) (AAAI 2025 · similarity 0.85)
• Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.84)