Research map / Large language models / AI and society
Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains two-layer neural network behavior models of how humans combine AI recommendations and LIME/SHAP explanations, then gradient-optimizes explanations to nudge human decisions toward adversarial (fairness-degrading) or benign (accuracy-improving) targets, validated with Prolific human experiments.
Problem
Whether human decisions in AI-assisted decision making can be quantitatively modeled from AI recommendations and explanations, and whether such models can be used to manipulate explanations to steer human decisions for adversarial or benign purposes, and whether people detect the manipulation.
Approach
Learn a neural behavior model H(x, y^m, e) of how humans use AI recommendations and explanations, then, without access to the AI model, gradient-optimize the explanation vector e' toward a targeted human decision under a sign-consistency constraint with the AI recommendation.
Key results
- Adversarially manipulated explanations significantly increased unfairness in human decisions (p<0.05 for FPRD in Census and Recidivism; p<0.01 for FNRD in Bias).
- Benignly manipulated explanations significantly improved human-AI team accuracy in Census, Recidivism, and Bias tasks, with no significant gain in Toxicity.
- Participants did not perceive significant differences in transparency, usefulness, or other perceptions between manipulated and original SHAP/LIME explanations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adversarially manipulated explanations instead of SHAP or LIME | Human-AI decision making · Census | FPRD (fairness) | Hurt |
| Use adversarially manipulated explanations instead of SHAP or LIME | Human-AI decision making · Recidivism | FPRD (fairness) | Hurt |
| Use adversarially manipulated explanations instead of SHAP or LIME | Human-AI decision making · Toxicity Detection | FPRD (fairness) | Hurt |
| Use adversarially manipulated explanations instead of SHAP or LIME | Human-AI decision making · Bias Detection | FNRD (fairness) | Hurt |
| Use adversarially manipulated explanations instead of SHAP or LIME | Human-AI decision making · Multiple tasks | Perceived transparency and usefulness | No clear effect |
| Use benignly manipulated explanations instead of SHAP or LIME | Human-AI decision making · Census | Accuracy | Helped |
| Use benignly manipulated explanations instead of SHAP or LIME | Human-AI decision making · Recidivism | Accuracy | Helped |
| Use benignly manipulated explanations instead of SHAP or LIME | Human-AI decision making · Bias Detection | Accuracy | Helped |
| Use benignly manipulated explanations instead of SHAP or LIME | Human-AI decision making · Toxicity Detection | Accuracy | No clear effect |
| Use benignly manipulated explanations instead of SHAP or LIME | Human-AI decision making · Multiple tasks | Perceived transparency and usefulness | No clear effect |
Surprising findings
- Participants were influenced by manipulated explanations yet generally failed to detect any anomalies in them.
- Behavior models were trained on only 42-80 participants per task, yet cross-validated accuracy reached 0.65-0.79.
Limitations
- Only score-based explanations (LIME, SHAP) were modeled and manipulated; example-based and rule-based explanations were not studied.
- Evaluated only on tabular and text tasks; image data not covered.
- Behavior models were trained on a small number of participants per task.
Experimental setup
Datasets: Census income Recidivism (COMPAS-style) Bias detection text dataset Toxicity detection text dataset
Benchmarks: Census Recidivism Bias Detection Toxicity Detection
Metrics: FPRD FNRD accuracy overreliance underreliance perceived transparency perceived usefulness
Baselines: SHAP explanations LIME explanations AI solo independent human decision human-AI combination method truth inference methods
Builds on: LIME SHAP BERT
Closest papers by meaning
- Decoding AI’s Nudge: A Unified Framework to Predict Human Behavior in AI-Assisted Decision Making (AAAI 2024 · similarity 0.93)
- Explanations are a Means to an End: Decision Theoretic Explanation Evaluation (ICML 2026 · similarity 0.91)
- Does Explainable Artificial Intelligence Improve Human Decision-Making? (AAAI 2021 · similarity 0.91)
- Learning to Decide with AI Assistance under Human-Alignment (NeurIPS 2026 · similarity 0.87)
- Mecha-nudges for Machines (NeurIPS 2026 · similarity 0.86)
- Using AI Uncertainty Quantification to Improve Human Decision-Making (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.