Research map / Reinforcement learning / Policy gradient and actor-critic
Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares switchback designs of varying switch frequency in Markovian environments under RL-based ATE estimators (model-based OLS, LSTD, DRL), showing that carryover effect size and reward error autocorrelation determine which design is most efficient.
Problem
Choosing the switchback design (switch frequency) for A/B tests with temporal carryover effects and autocorrelated rewards, where the ATE estimator's efficiency depends on the design and the estimator.
Approach
Derives a bias-variance-style MSE comparison between alternating-day and m-switchback designs for RL-based OPE estimators (model-based, LSTD, DRL) in finite MDPs with autocorrelated reward errors, showing that positively autocorrelated errors favor frequent switching when carryover is weak, while large carryover favors alternating-day designs.
Key results
- With positively autocorrelated reward errors and weak carryover, RMSE generally decreases as switch frequency increases (smaller m), with the gap versus alternating-day growing with the autocorrelation rho.
- With large carryover effect (larger delta), alternating-day design becomes progressively more efficient.
- Real-data simulation: with positively correlated residuals, frequent switching helps when effect size is 0 or 2%, while alternating-day is more efficient at 15% effect size; most CI coverage probabilities exceed 92%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use more frequent switchback alternations instead of alternating-day (AD) assignment | ATE estimation with RL estimators · Weak carryover, positively correlated reward errors | MSE | Helped |
| Use more frequent switchback alternations instead of alternating-day (AD) assignment | ATE estimation with RL estimators · Weak carryover, uncorrelated reward errors | MSE | No clear effect |
| Use alternating-day (AD) assignment instead of switchback assignment | ATE estimation with RL estimators · Weak carryover, majority negatively correlated reward errors | MSE | Helped |
| Use AD or switchback designs with less frequent switches instead of frequent switching | ATE estimation with RL estimators · Large carryover effect | MSE | Helped |
| Increase switch frequency | OLS, LSTD, and DRL estimators · Synthetic environments with weak carryover and positively correlated reward errors | RMSE | Helped |
| Increase carryover effect | OLS, LSTD, and DRL estimators · Synthetic environments | RMSE | Hurt |
| Use RL-based estimators instead of sequential IS and burn-in difference-in-mean estimators | OLS, LSTD, and DRL estimators · Synthetic linear and nonlinear DGPs | log(MSE) | Helped |
| Use LSTD or DRL instead of simple importance sampling | ATE estimation · Synthetic environments | log(MSE) | Helped |
| Use more frequent switchback alternations | ATE estimation · Real-data-based simulation with positively correlated reward residuals and no or minor effect enhancement | MSE | Helped |
| Use AD instead of frequent switchback alternations | ATE estimation · Real-data-based simulation with carryover increased to 15% | MSE | Helped |
| Use more frequent switching | OLS-based ATE estimator · Real-data-based simulation with small effect size | Confidence interval width | Helped |
| Use AD instead of switchback designs | OLS-based ATE estimator · Real-data-based simulation with effect size increased to 10% | Confidence interval width | Helped |
Surprising findings
- Off-policy switchback estimators can outperform on-policy alternating-day estimators in efficiency, contrary to the usual standard MDP intuition.
- With uncorrelated errors, AD generally performs best in finite samples even though all designs are asymptotically equivalent.
Limitations
- Theory relies on Markov assumption and bounded rewards; violations require redefining state or longer intervals.
- Large carryover effects make switchback designs suffer bias and distributional shift; not universally applicable (switching costs, complex interactions, POMDP settings).
- Real-data experiments rely on bootstrapped simulation from A/A data, with treatment effects manually constructed.
Experimental setup
Datasets: ridesharing company A/A experiment data (City I and City II), bootstrapped
Benchmarks: synthetic linear and nonlinear MDP environments real-data-based bootstrap simulation from ridesharing data
Metrics: root mean squared error mean squared error bias standard deviation coverage probability confidence interval width
Baselines: sequential importance sampling estimator burn-in difference-in-mean estimator simple importance sampling estimator
Builds on: double reinforcement learning LSTD marginal importance sampling double machine learning
Closest papers by meaning
- Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.86)
- Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation (ICML 2025 · similarity 0.85)
- Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.85)
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.85)
- Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime (NeurIPS 2026 · similarity 0.85)
- Policy Learning for Balancing Short-Term and Long-Term Rewards (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.