Research map / Reinforcement learning / Policy gradient and actor-critic
Step-DAD: Semi-Amortized Policy-Based Bayesian Experimental Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Step-DAD, a semi-amortized policy-based Bayesian experimental design method that fine-tunes a pretrained DAD policy mid-experiment using posterior samples from the data gathered so far.
Problem
Fully amortized policy-based Bayesian experimental design (e.g. DAD) learns a fixed design policy offline, which can be suboptimal and non-robust when real data deviates from simulated training histories or the prior is misspecified.
Approach
Decompose total EIG at an intermediate step so the remaining-steps policy depends only on the model and the current history; periodically infer the posterior from observed data and fine-tune a copy of the amortized DAD policy on the remaining-EIG sPCE bound, giving a semi-amortized infer-refine loop.
Key results
- On source location finding (T=10, tau=6), Step-DAD reaches total EIG lower bound 7.759 vs 7.040 for DAD; best tau=8 gives 7.877.
- On hyperbolic temporal discounting, Step-DAD (tau=10) achieves lower bound 6.711 vs 4.778 for DAD and 4.454 for Greedy (BADapted).
- Under increasing prior shift, DAD's EIG drops to essentially zero while Step-DAD retains positive information gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the fixed DAD policy with Step-DAD, fine-tuned at step $\tau=6$ | DAD policy for source location finding · Source Location Finding (T=10) | total EIG: 7.040 -> 7.759 | Helped |
| Use a 10K-step pretraining budget plus 2.5K fine-tuning steps rather than the 50K-step DAD baseline (ablation) | Source location finding policy · Source Location Finding | total EIG | Helped |
| Fine-tune the policy at different experiment steps $\tau$ (ablation) | Step-DAD source location finding policy · Source Location Finding (T=10) | total EIG: 7.050 ($\pm$ 0.103) at $\tau=1$; 7.877 ($\pm$ 0.082) at $\tau=8$ | Mixed |
| Increase the number of sources while keeping the update at $\tau=7$ (ablation) | Step-DAD · Source Location Finding, multiple sources | EIG difference over DAD: 0.701 $\pm$ 0.023; 0.426 $\pm$ 0.014; 0.423 $\pm$ 0.012 | Hurt |
| Fine-tune DAD at step $\tau$ | DAD policy for hyperbolic temporal discounting · Hyperbolic Temporal Discounting (T=20) | total EIG: 4.778 -> 6.711 | Helped |
| Fine-tune the policy at $\tau$ and $2\tau$ when extending the experiment to T=40 (ablation) | DAD policy for hyperbolic temporal discounting · Hyperbolic Temporal Discounting (T=40) | EIG: $3.9 \pm 0.24$ -> $\mathbf{4.8 \pm 0.14}$; $2.1 \pm 0.36$ -> $\mathbf{4.6 \pm 0.18}$ | Helped |
| Fine-tune DAD at step $\tau=5$ | DAD policy for CES · CES (T=10) | total EIG: $10.181 \pm 0.021$ -> $13.879 \pm 0.352$ | Helped |
| Reduce the inference budget from 100% to 20% (ablation) | Step-DAD · Source Location Finding | total EIG | Hurt |
| Increase the inference budget from 100% to 200% (ablation) | Step-DAD · Source Location Finding | total EIG | No clear effect |
| Perform more policy-update interventions (ablation) | Step-DAD · Source Location Finding | total EIG | Helped |
| Use 0.5K rather than 2.5K fine-tuning steps (ablation) | Step-DAD source location finding policy · Source Location Finding | total EIG | No clear effect |
Surprising findings
- Step-DAD with a reduced 10K-step DAD pretraining budget matches or exceeds DAD with the full 50K budget for tau>3 in location finding.
- On the CES model, the fully amortized DAD underperforms Step-Static, attributed to discontinuities in the censored likelihood causing convergence to local optima.
Limitations
- Requires some computation during the live experiment (posterior inference and policy fine-tuning), which may be unsuitable for very fast decisions.
- Posterior inference relies on importance sampling or other inference schemes whose cost and accuracy affect refinement quality.
- Gains are conservatively estimated via bounds, and the bounds are looser for some experiments (e.g. CES, temporal discounting).
Experimental setup
Datasets: simulated histories from source location finding model simulated histories from hyperbolic temporal discounting model simulated histories from CES model
Benchmarks: source location finding hyperbolic temporal discounting constant elasticity of substitution
Metrics: total expected information gain (lower and upper bounds) EIG under prior-perturbed test-time model
Baselines: DAD Static Step-Static Random Greedy (BADapted) Greedy (vPCE)
Builds on: Deep Adaptive Design iDAD sPCE sequential Nested Monte Carlo DeepSets
Closest papers by meaning
- Nesting Particle Filters for Experimental Design in Dynamical Systems (ICML 2024 · similarity 0.87)
- Amortized Bayesian Experimental Design for Decision-Making (NeurIPS 2024 · similarity 0.86)
- PASOA- PArticle baSed Bayesian Optimal Adaptive design (ICML 2024 · similarity 0.85)
- Maximin Robust Bayesian Experimental Design (NeurIPS 2026 · similarity 0.84)
- Bayesian Experimental Design Via Contrastive Diffusions (ICLR 2025 · similarity 0.83)
- On Estimating the Gradient of the Expected Information Gain in Bayesian Experimental Design (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.