Research map / Reinforcement learning / Offline RL
Off-Policy Selection for Initiating Human-Centric Experimental Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes First-Glance Off-Policy Selection (FPS), which clusters offline participants by initial state, augments each cluster with VAE-generated trajectories, and selects a per-cluster policy so new participants can be assigned policies from their initial state alone.
Problem
Off-policy selection for human-centric systems where participants are heterogeneous and a new participant arrives with only an initial state observed, without any prior offline trajectory for that participant.
Approach
Partition offline participants into sub-groups by initial-state features (via TICC clustering), augment each sub-group with a VAE, estimate per-sub-group value with an unbiased importance-sampling-based estimator with bounded variance, and deploy the per-sub-group best policy to arriving participants by matching their initial state to a cluster.
Key results
- In the IE experiment, FPS improved student NLG by 208% over the lecturer-designed behavioral policy and by 136% over the union of best existing-OPS baselines.
- In the sepsis simulator with N=2,500, FPS achieved absolute error 0.026 versus 0.054 for WIS and regret@1 0.042 versus 0.066 for WIS.
- FPS achieved the lowest AE and highest return across all three simulated dataset sizes (N=2,500, 5,000, 10,000).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use subgroup-specific policy selection rather than one population-wide policy | FPS · IE, 6th semester | NLG: 208% over the lecturer-designed behavioral policy | Helped |
| Use subgroup-specific policy selection rather than existing OPS-based selection | FPS · IE, 6th semester | NLG: 136% | Helped |
| Add trajectory augmentation (ablation) | FPS · IE, 6th semester | student performance | Helped |
| Use subgroup-specific policies instead of one policy for all participants (ablation) | FPS · IE, 6th semester | student performance | Helped |
| Add subgroup partitioning to OPS | Sepsis simulator, N=2,500 · Sepsis simulator, N=2,500 | Return: 0.121±0.01 -> 0.132±0.02 | Helped |
| Add subgroup partitioning to OPS | Sepsis simulator, N=5,000 · Sepsis simulator, N=5,000 | Return: 0.123±0.01 -> 0.149±0.01 | Helped |
| Add subgroup partitioning to OPS | Sepsis simulator, N=10,000 · Sepsis simulator, N=10,000 | Return: 0.129±0.00 -> 0.130±0.00 | Helped |
| Add subgroup partitioning (ablation) | FPS-noTA · Sepsis simulator, N=2,500 | AE: 0.054±0.00 -> 0.026±0.00 | Helped |
| Add subgroup partitioning (ablation) | FPS-noTA · Sepsis simulator, N=5,000 | AE: 0.046±0.00 -> 0.006±0.00 | Helped |
| Add subgroup partitioning (ablation) | FPS-noTA · Sepsis simulator, N=10,000 | AE: 0.022±0.00 -> 0.022±0.00 | No clear effect |
Surprising findings
- DualDICE could not distinguish returns across target policies in the IE experiment and was omitted from policy selection results.
- Existing OPS methods, even with VAE-based augmentation, still selected sub-optimal policies for most students in the IE experiment.
Limitations
- Real IE sub-groups are small (e.g., 24 to 678 training students per sub-group), limiting estimation reliability.
- Evaluated only as OPS; extension to offline RL policy optimization is left to future work.
- Proposed estimator uses IS weights, which can have high variance; bias-variance trade-offs with WDR or MAGIC are deferred.
Experimental setup
Datasets: IE system interaction logs from 1,288 students simulated sepsis trajectories
Benchmarks: real-world intelligent education experiment (6th semester cohort) sepsis simulator
Metrics: normalized learning gain absolute error return regret@1
Baselines: WIS PDIS FQE WDR MAGIC DualDICE OPS+RRS OPS+VRRS FPS-noTA FPS-P
Builds on: TICC variational auto-encoder for offline RL importance sampling fitted-Q evaluation critical-RL
Closest papers by meaning
- Get a Head Start: On-Demand Pedagogical Policy Selection in Intelligent Tutoring (AAAI 2024 · similarity 0.88)
- Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies (NeurIPS 2025 · similarity 0.85)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.85)
- Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination (ICML 2024 · similarity 0.85)
- EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes (NeurIPS 2024 · similarity 0.84)
- Towards Safe Policy Learning under Partial Identifiability: A Causal Approach (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.