AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

Beyond IPS: Reliable Counterfactual Evaluation in Multi-Stage Ad Systems without Logged Propensities

Mohsen Malmir, Mohamed A Radwan, houssam nassif, Murat Bayir

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Confounding-robust policy evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a counterfactual evaluation approach for offline assessment of ad ranking policies in multi-stage systems that works without logged propensities, addressing bias of Direct Method under distribution shift.

Problem

Offline evaluation of ad ranking policies in multi-stage delivery systems, where action propensities are often unavailable so IPS cannot be used, and Direct Method reward models can be biased when candidate policies induce impression distributions different from the logging policy.

Approach

Moves beyond IPS and plain Direct Method for multi-stage ad systems by estimating counterfactual value without logged propensities while correcting for impression distribution shift; abstract only describes the problem motivation, so specific mechanism is not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
reweight reward-model training toward the mixture of candidate-policy distributions (Weighted DM vs Unweighted DM) (ablation)Direct Method reward model · Ads simulator, Low ESS, Top-1 accuracyTop-1 accuracy: 0.67 -> 0.81Improved
mixture weighting at low overlap (ablation)Direct Method reward model · Ads simulator, Low ESS, Value MAE%Value MAE%: 7.20 -> 2.65Improved
mixture weighting at high overlap (ablation)Direct Method reward model · Ads simulator, High ESS, Value MAE%Value MAE%: 1.59 -> 1.86Hurt
use one shared mixture-weighted reward model instead of per-policy weighted models (ablation)Weighted DM · Ads simulator, Low ESS, Top-1 accuracyTop-1 accuracy: 0.75 -> 0.81Improved
use shared mixture-weighted model instead of per-policy models (ablation)Weighted DM · Production CTR rankingTop-1 accuracy: 0.24 -> 0.40Improved
add mixture weighting to Direct Method (ablation)Direct Method · Production CTR rankingTop-1 accuracy: 0.36 -> 0.40Improved
add doubly robust residual correction to mixture-weighted DM (ablation)Weighted DM · Ads simulator, Low ESS, Top-1 accuracyTop-1 accuracy: 0.81 -> 0.90Improved
use mixture-weighted reward model as DR control variate instead of unweighted (ablation)Doubly robust estimator · Ads simulator, Low ESS, Top-1 accuracyTop-1 accuracy: 0.82 -> 0.90Improved
use mixture-weighted reward model as DR control variate (ablation)Doubly robust estimator · Ads simulator, High ESS, Top-1 accuracyTop-1 accuracy: 1.00 -> 0.99No clear effect
use mixture-weighted reward model as DR control variate (ablation)Doubly robust estimator · Production CTR rankingTop-1 accuracy / Value MAE%: Acc 0.38 -> 0.44, MAE% 1.90 -> 2.04Mixed
use classifier-estimated IPS weights instead of reward-model estimatorsIPS · Production CTR rankingTop-1 regret: IPS 0.00807 vs Weighted DM 0.00244Hurt
clip density-ratio weights at c=1 (removes all up-weighting) (ablation)mixture weights · Ads simulator clipping sweepValue MAE%Hurt

Experimental setup

Baselines: inverse propensity scoring Direct Method

Builds on: inverse propensity scoring Direct Method

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy