Research map / Large language models / AI and society
Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audits eight Shapley value formulations via a unified amortized framework across five risk datasets and a 37-participant human study (3,735 case reviews), finding that quantitative XAI proxies do not predict human clarity or decision utility and that explanations raise confidence without improving accuracy.
Problem
Shapley-value formulations proliferate with divergent semantics of feature absence, yet evaluation relies on quantitative proxies (sparsity, faithfulness) whose alignment with human utility in high-stakes, low-latency workflows is unverified.
Approach
Uses a single amortized surrogate training pipeline (feed-forward amortizers with identical architecture and training) so that differences across Shapley value functions reflect semantics rather than approximation artifacts, then tests both functional proxies and analyst behavior in a controlled human-in-the-loop study.
Key results
- Across 37 participants and 3,735 case reviews, no Shapley formulation reliably improved decision accuracy or complex-case decision time, while Conditional, Counterfactual, and Mean formulations significantly inflated analyst confidence relative to no explanation.
- Joint-Marginal explanations significantly improved perceived clarity (odds ratio 1.52), while Zero, Uniform, and Filtered Conditional were rated as confusing.
- Quantitative proxies did not positively predict clarity or confidence; sparsity was negatively associated with confidence.
Surprising findings
- Sparsity, a metric assumed to aid interpretability, was negatively associated with analyst confidence.
- Explanations consistently boosted decision confidence while producing no measurable gain in objective accuracy, indicating automation-bias risk.
Limitations
- Restricted to low-latency tabular risk models; results may not transfer to vision or language.
- Controlled settings cannot capture long-term learning, adaptation, or institutional changes.
- Only five analysts among 37 participants were professionals, and the proprietary fraud dataset limits reproducibility of that part.
Experimental setup
Datasets: Maternal Risk German Credit Adult HELOC Fraud Risk (proprietary)
Benchmarks: Maternal Risk German Credit Adult HELOC Fraud Risk
Metrics: Deletion AUC Insertion AUC Perturbation sensitivity Counterfactual contrastivity Sparsity Attribution error Recall@k Cross-agreement (Spearman) Decision accuracy Decision time
Baselines: KernelSHAP Zero baseline Shapley Mean baseline Shapley Uniform Shapley Marginal Shapley Joint-Marginal Shapley Conditional Shapley Counterfactual Shapley Filtered Conditional Shapley no-explanation control
Builds on: SHAP KernelSHAP DiCE Optuna LightGBM
Closest papers by meaning
- X-Hacking: The Threat of Misguided AutoML (ICML 2025 · similarity 0.87)
- Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics (NeurIPS 2024 · similarity 0.86)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.85)
- EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting (NeurIPS 2026 · similarity 0.85)
- Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases (NeurIPS 2026 · similarity 0.84)
- Position: AI Safety Must Embrace an Antifragile Perspective (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.