Research map / Reinforcement learning / Offline RL
Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adaptive learn-then-test (aLTT), which uses e-processes and sequential adaptive acquisition with early termination to select reliable hyperparameters with finite-sample FWER/FDR guarantees using far fewer testing rounds than LTT.
Problem
Hyperparameter selection for AI apps with finite-sample statistical guarantees on population risk; standard learn-then-test (LTT) uses non-adaptive p-value-based testing and is data-inefficient, which matters when testing is costly or risky.
Approach
Replace p-values with e-values/e-processes so hypotheses can be tested sequentially with data-dependent acquisition (e.g., epsilon-greedy on e-process values) and anytime-valid p-values, allowing early termination while keeping FWER and FDR control.
Key results
- On HalfCheetah policy selection, aLTT with epsilon=0.25 raises TPR from 0.32 to 0.85 under FWER control and from 0.4 to 0.85 under FDR control, while LTT returns uninformative sets until the final round.
- In prompt engineering, aLTT with epsilon=0.25 identifies 50% of reliable instructions within the first 1000 rounds, versus under 10% for non-adaptive testing, and LTT's final TPR is less than half of aLTT's.
- aLTT consistently returns shorter selected instructions than LTT across accuracy requirements under a fixed budget of T=2000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use evidence-adaptive ε-greedy acquisition instead of non-adaptive random testing | aLTT for Half Cheetah policy selection · Half Cheetah, FWER control | TPR: 0.32 to 0.85 | Helped |
| Use evidence-adaptive ε-greedy acquisition instead of non-adaptive random testing | aLTT for Half Cheetah policy selection · Half Cheetah, FDR control | TPR: 0.4 to 0.85 | Helped |
| Use evidence-adaptive acquisition (ε = 0.25) instead of non-adaptive random testing | aLTT for prompt selection · Instruction induction tasks | TPR: 50% ... within the first 1000 testing rounds; ... less than 10% | Helped |
| Use evidence-adaptive acquisition instead of non-adaptive testing | aLTT for prompt selection · Instruction induction tasks | testing rounds to achieve a given TPR | Helped |
| Use aLTT instead of LTT for quantile-risk policy selection | TD3+BC policies · Offline RL quantile risk (q=0.1) | TPR: up to 75% | Helped |
| Use eBH FDR control instead of p-value-based BH for arbitrarily dependent tests | aLTT · Hyperparameter selection | FDR control validity | Helped |
| Use aLTT instead of LTT for wireless resource allocation selection | Wireless resource allocation policies · Queue occupancy and energy-efficiency requirements | energy-delay product | Helped |
| Use aLTT instead of LTT for wireless resource allocation selection | Wireless resource allocation policies · Energy-efficiency requirement | average delay | Helped |
| Use FDR control instead of FWER control | aLTT · Half Cheetah policy selection | TPR | Helped |
| Use FDR control instead of FWER control | aLTT for wireless delay selection · Wireless resource allocation | average delay | Helped |
| Replace aGRAPA betting with online Newton step betting (ablation) | aLTT for prompt selection · Instruction induction tasks | TPR | No clear effect |
| Replace aGRAPA betting with MaxBet (ablation) | aLTT for Half Cheetah policy selection · Half Cheetah | TPR | No clear effect |
Surprising findings
- Non-adaptive MaxBet betting strategy performs surprisingly well in the policy selection experiment, while UnitBet achieves the lowest TPR.
- The FDR-controlling procedure did not lose reliability in the wireless experiment: the FWER of the selected hyperparameter was well below delta=0.1.
Limitations
- Guarantees assume i.i.d. or stationary testing data; distribution shift is left to future work.
- Performance depends on the choice of acquisition policy and betting strategy (e.g., epsilon and aGRAPA).
Experimental setup
Datasets: OpenAI Gym MuJoCo HalfCheetah offline data instruction induction dataset Nokia Wireless Suite simulator
Benchmarks: HalfCheetah online policy selection instruction induction prompt selection wireless resource allocation simulation
Metrics: true positive rate FWER FDR number of testing rounds instruction length energy-delay product
Baselines: LTT (non-adaptive learn-then-test) aLTT with non-adaptive acquisition
Builds on: learn-then-test e-value multiple hypothesis testing aGRAPA betting strategy e-Benjamini-Hochberg
Closest papers by meaning
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.86)
- Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments (ICML 2025 · similarity 0.85)
- Optimizing Test-Time Compute via Meta Reinforcement Finetuning (ICML 2025 · similarity 0.85)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.