Research map / Reinforcement learning / Offline RL
Offline Model-Based Optimization via Policy-Guided Gradient Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Policy-guided Gradient Search (PGS), which reformulates offline model-based optimization as offline RL that learns a step-size policy guiding gradient ascent on a learned surrogate, trained on random top-p trajectories with CQL.
Problem
Offline black-box optimization: finding high-value inputs outside a fixed offline dataset when only a learned surrogate is available, where naive surrogate optimization overestimates out-of-distribution inputs and fixed gradient search strategies are suboptimal.
Approach
Cast the search strategy itself as a learned policy: the policy outputs a per-dimension step-size vector for gradient ascent on a surrogate, and the policy is learned with offline RL (CQL) on random trajectories synthesized from the top-p percentile of the offline data, with OSEL latent-embedding KNN estimates used for hyperparameter selection.
Key results
- PGS achieves the best scores on Superconductor (0.563 ± 0.058), D'Kitty Morphology (0.966 ± 0.013) and TF Bind 8 (0.981 ± 0.015), with an average rank of 2.16/13 across tasks, the best among compared methods.
- Replacing offline RL (CQL) with standard soft actor-critic in PGS lowers performance, e.g., Ant Morphology drops from 0.949 to 0.303.
- Top-p random trajectories outperform monotonic trajectories (BONET-style) on Ant, D'Kitty and TFBind8 (e.g., TFBind8 0.981 vs 0.925).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use offline RL rather than standard RL to train the policy (ablation) | PGS · Superconductor | normalized objective: 0.528 ± 0.012 -> 0.563 ± 0.058 | Helped |
| use offline RL rather than standard RL to train the policy (ablation) | PGS · Ant Morphology | normalized objective: 0.303 ± 0.017 -> 0.949 ± 0.017 | Helped |
| use offline RL rather than standard RL to train the policy (ablation) | PGS · D'Kitty Morphology | normalized objective: 0.900 ± 0.007 -> 0.966 ± 0.013 | Helped |
| use offline RL rather than standard RL to train the policy (ablation) | PGS · GFP | normalized objective: 0.864 ± 0.000 -> 0.864 ± 0.000 | No clear effect |
| use offline RL rather than standard RL to train the policy (ablation) | PGS · TF Bind8 | normalized objective: 0.774 ± 0.063 -> 0.981 ± 0.015 | Helped |
| use offline RL rather than standard RL to train the policy (ablation) | PGS · UTR | normalized objective: 0.686 ± 0.013 -> 0.713 ± 0.009 | Helped |
| use top-p subset rather than the entire offline dataset for trajectories (ablation) | PGS · Ant Morphology | normalized objective: 0.890 ± 0.022 -> 0.949 ± 0.017 | Helped |
| use top-p subset rather than the entire offline dataset for trajectories (ablation) | PGS · D'Kitty Morphology | normalized objective: 0.947 ± 0.013 -> 0.966 ± 0.013 | Helped |
| use top-p subset rather than the entire offline dataset for trajectories (ablation) | PGS · Superconductor | normalized objective: 0.533 ± 0.036 -> 0.563 ± 0.058 | Helped |
| use top-p subset rather than the entire offline dataset for trajectories (ablation) | PGS · TF Bind8 | normalized objective: 0.955 ± 0.024 -> 0.981 ± 0.015 | Helped |
| use random top-p trajectories rather than monotonic trajectories (ablation) | PGS · Ant Morphology | normalized objective: 0.916 ± 0.027 -> 0.949 ± 0.017 | Helped |
| use random top-p trajectories rather than monotonic trajectories (ablation) | PGS · D'Kitty Morphology | normalized objective: 0.964 ± 0.013 -> 0.966 ± 0.013 | No clear effect |
Surprising findings
- Standard RL without conservatism collapses on Ant Morphology (0.303), far below the surrogate-gradient baseline Grad (0.293 is similar, but PGS w/ standard RL is much worse than PGS with offline RL).
- Performance of PGS decreases as test-time gradient search steps increase beyond the 50 used in training on most tasks, attributed to train-test mismatch.
Limitations
- Performance depends on state-space coverage of the offline dataset; may degrade on poorly covered data such as Ant Morphology.
- Hyperparameter selection (top-p and CQL epochs) relies on the OSEL proxy, and the paper notes finding good hyperparameters remains open.
- Test-time performance degrades with more search steps than the 50 used in training, indicating train-test mismatch.
Experimental setup
Datasets: D'Kitty Morphology (design-bench) Ant Morphology (design-bench) Superconductor (design-bench) GFP (design-bench) TF Bind 8 (design-bench) UTR (design-bench)
Benchmarks: D'Kitty Morphology Ant Morphology Superconductor GFP TF Bind 8 UTR
Metrics: 100th percentile normalized objective score mean rank across tasks
Baselines: BO-qEI CMA-ES REINFORCE CbAS Auto.CbAS MIN BONET Grad COMs ROMA NEMO BDI
Builds on: conservative Q-learning design-bench BONET implicit Q-learning beta-VAE
Closest papers by meaning
- Learning Surrogates for Offline Black-Box Optimization via Gradient Matching (ICML 2024 · similarity 0.92)
- Boosting Offline Optimizers with Surrogate Sensitivity (ICML 2024 · similarity 0.91)
- Incorporating Surrogate Gradient Norm to Improve Offline Optimization Techniques (NeurIPS 2024 · similarity 0.91)
- ROOT: Rethinking Offline Optimization as Distributional Translation via Probabilistic Bridge (NeurIPS 2025 · similarity 0.88)
- Generative Adversarial Model-Based Optimization via Source Critic Regularization (NeurIPS 2024 · similarity 0.88)
- Offline Model-Based Optimization by Learning to Rank (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.