Research map / Probabilistic and scientific modeling / Treatment effects
On Online Experimentation without Device Identifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HIFIVE, a variational latent-variable method that estimates global average treatment effects in online A/B tests when device identifiers are missing, using only a superset of each user's devices.
Problem
Estimating global average treatment effects in randomized experiments when users' devices cannot be linked, so device-level outcomes suffer interference from other devices of the same user and the device graph is uncertain.
Approach
Treats the true exposure (based on the unknown device-user graph) as a latent variable observed only through a noisy exposure computed from a superset of neighbours, and fits a variational importance-weighted model with neural-network outcome functions; proves identifiability of the conditional mean via noisy-regression results.
Key results
- On Erdos-Renyi synthetic graphs HIFIVE yields near-unbiased GATE estimates while Poly, ReFeX, PERC and DM are highly biased under interference.
- On the AirBnB simulator HIFIVE performs comparably to the oracle-graph exposure (EXP) model in bias and MSE without access to the true graph.
- On power plant emissions data HIFIVE gives estimates similar to EXP and ReFeX without exact neighbourhood information.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HIFIVE instead of standard estimators | GATE estimation on synthetic Erdős–Rényi graphs · Synthetic Erdős–Rényi graphs, linear and nonlinear outcomes | relative bias | Helped |
| Increase interference strength | HIFIVE and baseline estimators · Synthetic Erdős–Rényi graphs | relative bias | Mixed |
| Increase population size | HIFIVE · Synthetic Erdős–Rényi graphs | variance | Helped |
| Increase treatment probability toward 0.5 | HIFIVE · Synthetic Erdős–Rényi graphs | variance | Helped |
| Use HIFIVE instead of the difference-in-means estimator | GATE estimation · AirBnB simulation | absolute relative bias | Helped |
| Use HIFIVE with uncertain neighborhoods rather than the oracle exposure model | GATE estimation · AirBnB simulation | relative RMSE | No clear effect |
| Add extraneous neighbors while retaining a superset of true neighbors | HIFIVE · Synthetic Erdős–Rényi graphs and AirBnB simulation | absolute relative bias | No clear effect |
| Increase the number of extraneous neighbors | HIFIVE · Synthetic Erdős–Rényi graphs and AirBnB simulation | uncertainty in estimation | Hurt |
| Omit pertinent neighbors from the assumed neighborhood | HIFIVE · Synthetic Erdős–Rényi graphs and AirBnB simulation | absolute relative bias | Hurt |
| Use HIFIVE rather than methods requiring exact neighborhoods | GATE estimation of power-plant pollution controls · Power-plant emissions data | concentration change | No clear effect |
Surprising findings
- Bias decreases but variance increases as the assumed neighbourhood superset grows, so larger neighbourhoods trade bias for uncertainty.
Limitations
- Variance of the estimate grows with the size of the assumed neighbourhoods.
- Identifiability does not hold when the assumed neighbourhood is a strict subset of true neighbours.
- Relies on a superset assumption on device neighbourhoods and on exposure-model assumptions that the AirBnB experiment shows can be violated.
Experimental setup
Datasets: synthetic Erdos-Renyi graphs AirBnB simulator US power plant emissions observational dataset (Papadogeorgou et al.)
Benchmarks: synthetic Erdos-Renyi networks AirBnB simulator power plant emissions data
Metrics: relative bias relative RMSE variance
Baselines: difference-in-means (SUTVA) estimator Horvitz-Thompson estimator polynomial regression ReFeX PERC/DWR EXP exposure model
Builds on: exposure mapping importance weighted autoencoder DReG estimator Horvitz-Thompson estimator
Closest papers by meaning
- Causal Inference out of Control: Estimating Performativity without Treatment Randomization (ICML 2024 · similarity 0.87)
- Transferring Causal Effects using Proxies (NeurIPS 2025 · similarity 0.86)
- Online Experimental Design With Estimation-Regret Trade-off Under Network Interference (NeurIPS 2025 · similarity 0.85)
- Higher-Order Causal Message Passing for Experimentation with Complex Interference (NeurIPS 2024 · similarity 0.85)
- Telling Peer Direct Effects from Indirect Effects in Observational Network Data (ICML 2025 · similarity 0.85)
- Active Treatment Effect Estimation via Limited Samples (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.