Research map / Large language models / AI and society

Bridging the Knowledge Gap: Understanding User Expectations for Trustworthy LLM Standards

AAAI 2025 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Mixed-method study (12 interviews, 379-person Prolific survey) that elicits lay users' individual criteria for high-level trustworthy AI standards for LLMs and how they prioritize them.

Problem

Experts who define trustworthy AI standards for LLMs and the users whose values are affected have a knowledge gap; users lack information to judge whether standards are met, and existing top-down standards are abstract.

Approach

A bottom-up approach that elicits user-defined individual standards (68 criteria) under 13 high-level standards and ranks them via interviews and pairwise-comparison survey, then proposes a search/experience/credence taxonomy for communicating standards.

Key results

• Qualitative interviews identified 68 individual standards across 13 high-level standards, ranging from 4 (Accountability, Robustness) to 8 (User Experience) per category.
• Survey participants rated privacy (M=87.3), safety (M=86.6) and security (M=86.0) as most important, and sustainability lowest (M=61.4).
• Users favored data protection through login over transparent information provision for privacy, and placed more value on transparent data-handling information than on system or output information.

Surprising findings

• Users showed knowledge gaps on accountability and auditability and were largely uncertain about them.
• Most identified standards were judged not effectively communicable through typical technical approaches such as model output explanations.

Limitations

• Qualitative interview sample (N=12) is potentially non-representative; demographic differences not examined.
• Survey only compared individual standards within a single high-level standard.
• Focus limited to conversational LLM applications.

Experimental setup

Datasets: interview data (N=12) Prolific online survey responses (N=379)

Metrics: mean importance rating (1-100) pairwise preference percentages

Builds on: Schlicker et al. 2023 individual standards (TrAM) Liao and Vaughan 2023 AI transparency roadmap search, experience, and credence framework

Closest papers by meaning

• Position: TrustLLM: Trustworthiness in Large Language Models (ICML 2024 · similarity 0.87)
• ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.86)
• Natural Language Generation with Expert Standards (AAAI 2025 · similarity 0.86)
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.86)
• Position: Beyond Assistance – Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Care (ICML 2025 · similarity 0.85)
• Position: Trustworthy AI Agents Require the Integration of Large Language Models and Formal Methods (ICML 2025 · similarity 0.85)