Research map / Probabilistic and scientific modeling / Bayesian optimization
Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Experts-as-Priors (ExPerior), an empirical Bayes method that learns a prior over unobserved contexts from expert demonstrations (parametric or max-entropy) and uses posterior sampling for online decision-making in bandits, MDPs, and POMDPs.
Problem
Online sequential decision-making when auxiliary expert demonstrations were made using contextual information unobserved by the learner, so the expert data comes from related but different tasks.
Approach
Treat unobserved contexts as Bayesian parameters, learn an informative prior from expert trajectories by maximum marginal likelihood (parametric) or maximum entropy subject to a KL-constrained fit to expert data (nonparametric), then run posterior sampling (SGLD or bootstrapped DQN) online.
Key results
- On K=10 Bernoulli bandits, ExPerior-MaxEnt performs comparably to the SGLD variant of Oracle-TS, and both ExPerior variants beat Naive-TS, Naive-UCB, UCB-ExPLORe and BC in low- and mid-entropy settings.
- In Deep Sea (M=30), ExPerior outperforms Naive Boot-DQN and ExPLORe in all four goal distributions, with the gap shrinking as context entropy increases.
- In Frozen Lake (POMDP), ExPerior-MaxEnt reaches 11.47 average reward at beta=2.5 with 9 hazards versus about -23 for Naive Boot-DQN, though ExPLORe is better at beta=0.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use expert demonstrations with ExPerior rather than ExPLORe | Frozen Lake POMDP · Frozen Lake POMDP, fixed β = 1, # Hazard = 2 | average reward per episode: -30.68 ± 12.40 -> 11.47 ± 0.52 | Helped |
| Use ExPerior rather than ExPLORe | Frozen Lake MDP · Frozen Lake MDP, fixed β = 1, # Hazard = 2 | average reward per episode: -113.84 ± 17.50 -> 13.02 ± 0.18 | Helped |
| Use ExPerior rather than online Naïve Boot-DQN | Deep Sea · Deep Sea, goal distributions of varying entropy | average reward per episode | Helped |
| Use ExPerior rather than Naïve-TS or Naïve-UCB | K-armed Bernoulli bandits · Low-entropy optimal actions | Bayesian regret | Helped |
| Increase entropy of the optimal action | ExPerior · Bernoulli bandits | Bayesian regret | Hurt |
| Train ExPerior-MaxEnt with β = 10 across expert types (ablation) | ExPerior-MaxEnt · Bernoulli bandits, misspecified expert models | Bayesian regret: 51.7 ± 5.1 -> 29.7 ± 3.6 | Helped |
| Train ExPerior-Param with β = 1 across expert types (ablation) | ExPerior-Param · Bernoulli bandits, misspecified expert models | Bayesian regret: 24.3 ± 1.2 -> 8.6 ± 2.7 | Helped |
| Replace the Beta prior with a Normal prior (ablation) | ExPerior-Param · Bernoulli bandits, low entropy | Bayesian regret: 0.7 ± 0.3 -> 60.2 ± 6.3 | Hurt |
| Replace the Beta prior with a Gamma prior (ablation) | ExPerior-Param · Bernoulli bandits, high entropy | Bayesian regret: 24.5 ± 2.8 -> 51.8 ± 3.6 | Hurt |
| Use MaxEnt instead of a Beta parametric prior (ablation) | ExPerior · Bernoulli bandits, low entropy | Bayesian regret: 0.7 ± 0.3 -> 11.6 ± 1.3 | Hurt |
| Use ExPerior-MaxEnt rather than ExPerior-Param (ablation) | Frozen Lake POMDP · Frozen Lake POMDP, fixed # Hazard = 9, β = 1 | average reward per episode: -4.31 ± 1.80 -> 6.00 ± 0.00 | Helped |
| Use ExPerior-Param rather than ExPerior-MaxEnt (ablation) | Frozen Lake POMDP · Frozen Lake POMDP, fixed β = 1, # Hazard = 2 | average reward per episode: 11.47 ± 0.52 -> 12.00 ± 0.37 | Helped |
Surprising findings
- Bayesian regret of ExPerior scales approximately linearly with the entropy of the optimal action, in contrast to the sublinear O(sqrt(Ent)) dependence of standard Thompson sampling bounds.
- ExPLORe outperforms ExPerior in Frozen Lake when expert data is nearly random (beta=0.1), attributed to its actor-critic backbone.
Limitations
- Limited set of experiments, with no human-in-the-loop evaluation.
- Transition functions assumed invariant to context in MDPs, and the Bootstrapped DQN heuristic replaces exact Langevin posterior sampling.
- Regret bound for the frequentist successive elimination variant is worse than standard by a factor of K.
Experimental setup
Datasets: synthetic Bernoulli bandit expert demonstrations Deep Sea expert demonstrations (synthetic) Frozen Lake expert demonstrations (synthetic)
Benchmarks: K-armed Bernoulli bandits Deep Sea Frozen Lake
Metrics: Bayesian regret average reward per episode
Baselines: Behaviour cloning Naive-TS Naive-UCB UCB-ExPLORe Oracle-TS Naive Boot-DQN ExPLORe
Builds on: Thompson sampling ExPLORe bootstrapped DQN maximum marginal likelihood successive elimination
Closest papers by meaning
- Causal Imitation for Markov Decision Processes: a Partial Identification Approach (NeurIPS 2024 · similarity 0.89)
- Demonstration-Regularized RL (ICLR 2024 · similarity 0.87)
- Diffusion Models Meet Contextual Bandits (NeurIPS 2025 · similarity 0.87)
- Bayesian Decision Making around Experts (NeurIPS 2026 · similarity 0.87)
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.87)
- Generalization to New Sequential Decision Making Tasks with In-Context Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.