Research map / Large language models / AI and society
Position: Explainable AI Cannot Advance Without Better User Studies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that user studies are essential for evaluating explainable AI and that current XAI user studies are of poor quality, supported by a review and an analysis of 607 XAI papers with user studies.
Problem
User studies in explainable AI are rare, low quality, and low fidelity to real-world use, so the field lacks understanding of whether XAI methods actually satisfy societal desiderata.
Approach
Argues that because XAI's end goal is satisfying societal desiderata that require humans to measure, user studies are indispensable; it quantifies methodological quality (reproducibility, baselines, placebo controls, limitations, fidelity) across 607 papers to show the gap, and calls for higher standards and less reliance on functional evaluation.
Key results
- Very few user studies from top-4 venues (NeurIPS, ICLR, ICML, AAAI) contain enough information to be reproducible, versus an estimated 50%-90% for other papers.
- Placebo studies are rare; baselines are included in about half of studies, and validated questionnaires in less than half.
- Most studies are medium-fidelity downstream performance or low-fidelity understanding studies; there are very few high-fidelity studies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use subjective explanation-quality ratings as a proxy for explanation effectiveness | XAI evaluation · simulation tests | prediction of explanation effectiveness | No clear effect |
| use proxy tasks instead of actual decision-making tasks | XAI evaluation · actual decision-making tasks | prediction of actual task results | No clear effect |
| use subjective measures of trust and preference to predict performance | XAI evaluation · actual decision-making tasks | prediction of objective performance | No clear effect |
| present placebo explanations rather than real explanations | XAI explanations · perceived trust | perceived trust | No clear effect |
| describe a system as having AI | sham systems · subjective measurements | subjective measurements | Helped |
| describe a system as having AI | sham systems · objective measurements | objective measurements | No clear effect |
| closely reflect the deployment context | XAI user studies · real-world decision-making | practical utility of explanations | Hurt |
| rely on self-reported metrics rather than decision outcomes | XAI evaluation · decision-making | alignment with decision-making improvement | No clear effect |
Surprising findings
- Very few user studies from top ML venues (NeurIPS, ICLR, ICML, AAAI) contain enough information to be considered reproducible.
- No XAI user study was found that presents a novel explanation against a placebo-control group.
Limitations
- Literature sample is biased by keyword search and Scopus coverage; the analysis is exploratory with a single rater per variable.
- Sample biased toward older papers from indirect search and limited venue coverage.
- Position is not backed by confirmatory study; claims about the field rest on a sample estimated to be about half of all XAI user-study papers.
Experimental setup
Datasets: 607 XAI papers with user studies (Scopus-based collection)
Metrics: reproducibility rate baseline inclusion rate placebo study frequency participant count distribution fidelity level distribution evaluation type frequency
Builds on: Doshi-Velez & Kim taxonomy Nauta et al. 2023 XAI evaluation survey Speith & Langer evaluation model Kosch et al. 2023 placebo HCI study taxonomy
Closest papers by meaning
- Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics (NeurIPS 2024 · similarity 0.88)
- Position: Explain to Question not to Justify (ICML 2024 · similarity 0.88)
- Evaluating the Effectiveness of Explainable Artificial Intelligence Approaches (Student Abstract) (AAAI 2024 · similarity 0.85)
- Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (ICLR 2025 · similarity 0.85)
- Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary (NeurIPS 2024 · similarity 0.85)
- iLLuMinaTE: An LLM-XAI Framework Leveraging Social Science Explanation Theories Towards Actionable Student Performance Feedback (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.