Research map / Large language models / Language, safety and interpretability
On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-runs a prior e-commerce fraud-detection user study of post-hoc explainers (LIME, TreeSHAP, TreeInterpreter) with relaxed simplifying assumptions (real base rate, dollar-based regret metric, escalation option, placebo explanation arms), finding no utility for explanations and slower decisions.
Problem
Existing evaluations of explainable ML methods use simplified tasks, proxy users, subjective metrics, and experimental designs that depart from the deployment context, so they likely overestimate real-world utility of explanations.
Approach
Redesign an earlier fraud-analyst study to match the deployment setting (real 15% base rate, a Percent Dollar Regret metric reflecting business costs, an escalate option, larger samples, placebo random/irrelevant explanation arms, uniform post-decision questions, and analyst interviews) and show that conclusions reverse relative to the prior study.
Key results
- ML model score made analysts significantly faster (~30 s less decision time, p=.0001) and improved accuracy by 6pp, but PDR showed no appreciable improvement.
- No tested explanation method (LIME, TreeSHAP, TreeInterpreter) improved PDR or confusion-matrix metrics over the ML model; TreeSHAP and TreeInterpreter slowed decisions by ~16 s and ~14 s respectively.
- Random and irrelevant placebo explanations produced outcomes similar to real explanations, while explanations raised self-reported confidence, a mismatch with objective outcomes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Show the ML fraud score alongside transaction data | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | decision time: 79.2 -> 51.3; β = −27.9 | Helped |
| Show the ML fraud score alongside transaction data | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | accuracy: 76.6 -> 82.2; 6pp | Helped |
| Show the ML fraud score alongside transaction data | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | PDR: 9.5 -> 8.9 | No clear effect |
| Add TreeSHAP explanations to the transaction data and model score | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | decision time: 51.3 -> 67.3; ∼16 seconds | Hurt |
| Add TreeInterpreter explanations to the transaction data and model score | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | decision time: 51.3 -> 65.6; ∼14 seconds | Hurt |
| Add post-hoc explanations (TreeSHAP, TreeInterpreter, or LIME) to the model score | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | PDR: Model 8.9; TreeSHAP 9.7; TreeInterpreter 10; LIME 11.6 | No clear effect |
| Add LIME explanations to the transaction data and model score | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | decision time: 51.3 -> 57.7 | No clear effect |
| Show random-feature explanations with the model score | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | PDR: 8.9 -> 9 | No clear effect |
| Show irrelevant explanations with the model score | Fraud analysts reviewing e-commerce transactions · E-commerce fraud detection | decision time: 51.3 -> 62.2 | Hurt |
| Use a domain-specific PDR metric instead of confusion-matrix accuracy | Evaluation of the ML model score for fraud analysts · E-commerce fraud detection | PDR | No clear effect |
Surprising findings
- Explanations made analysts slower without improving PDR, reversing the earlier study's accuracy-based conclusion of utility.
- Random and irrelevant explanations performed similarly to real explanation methods on objective metrics.
Limitations
- Only three professional analysts were available, so randomization was at the transaction level rather than the user level.
- Analysts had no opportunity for intermediate feedback during short experimental arms, unlike real deployment.
- Study limited to one merchant and one fraud detection context, with PDR parameters set from data analysis and analyst consultation.
Experimental setup
Datasets: 231,362 historical e-commerce transactions from a large merchant (Sep-Nov 2019)
Metrics: Percent Dollar Regret (PDR) mean decision time accuracy FPR TPR precision escalation rate self-reported confidence perceived information quality
Baselines: Data only ML model score only LIME TreeSHAP TreeInterpreter Random explanations Irrelevant explanations
Builds on: Jesus et al. 2021 fraud explanation study LIME TreeSHAP TreeInterpreter Amarasinghe et al. 2020 evaluation framework
Closest papers by meaning
- Position: Rethinking Explainable Machine Learning as Applied Statistics (ICML 2025 · similarity 0.85)
- Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings (NeurIPS 2026 · similarity 0.84)
- Auditing Local Explanations is Hard (NeurIPS 2024 · similarity 0.83)
- ScamNet: Toward Explainable Large Language Model-Based Fraudulent Shopping Website Detection (AAAI 2025 · similarity 0.83)
- Position: Explainable AI Cannot Advance Without Better User Studies (ICML 2025 · similarity 0.83)
- Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.