Research map / Large language models / AI and society
HearSayBench: Can LLMs Navigate from Abstract Human Rights to Lived Lives?
Sobhan Lotfi, Ava Iranmanesh, Ali Iranmanesh, Liwei Jiang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HearSayBench, a benchmark probing whether LLMs can translate abstract human rights into advice suited to users' lived constraints such as war or statelessness.
Problem
LLM advisors on life-critical problems implicitly assume users have full personal agency (the internet reflects people free to act), so existing benchmarks miss users in war zones or stateless situations.
Approach
A benchmark framing that tests whether LLMs can bridge abstract human-rights principles and the concrete constraints of people with limited agency, rather than assuming the freedom to act.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a stronger model within a family (Claude-Sonnet-4.6 to Claude-Opus-4.8) | Claude models · HearsayBench | Capability: 2.92 -> 3.63 | Improved |
| Use a stronger model within a family (Claude-Sonnet-4.6 to Claude-Opus-4.8) | Claude models · HearsayBench | Harm average: 2.95 -> 2.86 | Hurt |
| Use a stronger Gemini model (Gemini-3-Flash to Gemini-3.1-Pro) | Gemini models · HearsayBench | Harm average: 2.80 -> 2.91 | No clear effect |
| Use a stronger Gemini model (Gemini-3-Flash to Gemini-3.5-Flash) | Gemini models · HearsayBench | Harm average: 2.80 -> 2.85 | No clear effect |
| Use a stronger Gemini model (Gemini-3-Flash to Gemini-3.1-Pro) | Gemini models · HearsayBench | Capability: 3.02 -> 3.63 | Improved |
| Use open-weight instead of proprietary models | LLM advisors · HearsayBench | Comprehension-Capability gap: 19% vs 30% | Hurt |
| Use stronger models (top three vs weakest four) | LLM advisors · HearsayBench | Comprehension-Capability gap: 12 to 13% vs 28 to 37% | Improved |
| Use stronger models (higher Weighted Overall) | LLM advisors · HearsayBench | Tone minus Harm average gap: r = 0.96 | Hurt |
| Score Capability instead of Comprehension for the same model | All 14 models · HearsayBench | score (1-5): 0.85 points average drop | Hurt |
| Hand-written prompts instead of LLM-generated prompts (ablation) | Benchmark construction · HearsayBench scenarios | prompt quality (qualitative) | Improved |
| Condition scenarios on Power Relations vs other subtypes | LLM advisors · HearsayBench | Harm average: 2.05 (physical harm) | Hurt |
Experimental setup
Benchmarks: HearSayBench
Closest papers by meaning
- Position: Beyond Assistance – Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Care (ICML 2025 · similarity 0.86)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.83)
- Bridging the Knowledge Gap: Understanding User Expectations for Trustworthy LLM Standards (AAAI 2025 · similarity 0.83)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.83)
- ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.83)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
