Research map / Multi-agent systems and games / Games and equilibria
Strategic Hypothesis Testing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models hypothesis testing (p-value threshold choice) as a Stackelberg game with a strategic agent who chooses participation and sample size, and characterizes the principal's optimal p-value via a critical threshold, validated on public drug-approval cost/revenue data.
Problem
A principal (e.g., FDA) picks a p-value threshold for approving products while strategic agents decide whether to participate and how many samples to collect; the principal wants to balance Type I and Type II errors under this strategic response.
Approach
Extends prior Stackelberg hypothesis-testing work by letting agents choose sample size with per-sample cost and fixed cost, and by penalizing both false positives and false negatives; shows loss components are monotone piecewise around a critical p-value where the participation threshold equals the baseline, and that the optimal threshold is at least this critical value.
Key results
- Shows the principal's false positive loss is zero below the critical p-value and non-decreasing above it, while overall false negative loss is non-increasing in alpha.
- Critical p-value can be computed to epsilon accuracy in O(log^2(1/eps) log(n_max)) time.
- On drug-approval case study, the conventional p-value of 0.05 aligns roughly with median revenue thresholds for oncology and cardiovascular drugs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the p-value threshold up to the critical threshold α̂ | Strategic hypothesis testing · Products with effectiveness distribution q | false positive rate | No clear effect |
| Increase the p-value threshold below α̂ | Strategic hypothesis testing · Effective products (μ₀ ≥ μ_b) | false negative rate | Helped |
| Increase the p-value threshold above α̂ | Strategic hypothesis testing · Products with effectiveness distribution q | false positive rate | Hurt |
| Increase the p-value threshold above α̂ | Strategic hypothesis testing · Effective products (μ₀ ≥ μ_b) | false negative rate | Helped |
| Increase the p-value threshold below α̂ | Strategic hypothesis testing · Effective agents who participate | participating-agent false negative rate | Mixed |
| Increase the p-value threshold | Strategic hypothesis testing · Agent participation | participation threshold μτ(α) | Helped |
| Increase the p-value threshold | Strategic hypothesis testing · Effective agents who participate | pass probability | Helped |
| Set the threshold to α̂ rather than a lower threshold | Strategic hypothesis testing · Products with effectiveness distribution q | principal loss | Helped |
| Use α = 0.05 | Strategic drug-approval model · Oncology, cardiovascular, and vaccine drug categories | critical p-value threshold: 0.05 | No clear effect |
| Set revenue around $5 billion | Strategic drug-approval model · Oncology | critical p-value threshold: 0.1 and 0.2 | Helped |
Surprising findings
- The false negative rate among participating agents is not monotone in the p-value: it can increase then decrease, because raising alpha draws in low-effect agents who pass with low probability.
- Raising the p-value can paradoxically reduce pass probability when marginal sample costs rise faster than revenue gains.
Limitations
- Relies on a normal approximation to the binomial test and a Bayesian prior over effectiveness known to the principal.
- Drug cost and revenue inputs are rough, drawn from heterogeneous public sources, and the per-sample cost is fixed rather than estimated.
Experimental setup
Datasets: publicly available drug approval cost and revenue data (oncology, cardiovascular, vaccines)
Metrics: critical p-value threshold vs revenue false positive and false negative losses
Builds on: Strategic hypothesis testing Stackelberg game (prior work) Bayesian Stackelberg game
Closest papers by meaning
- Optimizing Social Utility in Sequential Experiments (NeurIPS 2026 · similarity 0.83)
- A Cramér–von Mises Approach to Incentivizing Truthful Data Sharing (NeurIPS 2025 · similarity 0.83)
- Causal Inference from Competing Treatments (ICML 2024 · similarity 0.83)
- Who’s Gaming the System? A Causally-Motivated Approach for Detecting Strategic Adaptation (NeurIPS 2024 · similarity 0.83)
- Causal Strategic Learning with Competitive Selection (AAAI 2024 · similarity 0.83)
- Private Blotto: Viewpoint Competition with Polarized Agents (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.