Research map / Reinforcement learning / Visual and representation RL
Searching for High-Value Molecules Using Reinforcement Learning and Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of text-based RL (REINFORCE over SMILES/SELFIES with pretrained FC, RNN, and GPT-style transformer policies) for molecular design, yielding the ChemRLformer recipe evaluated on 25+ docking and PyTDC oracle tasks.
Problem
Finding high-value molecules with text-based reinforcement learning requires careful choice of molecular text grammar, pretraining data, architecture, and RL algorithm design, which are not well understood.
Approach
Systematically ablate text grammar (SMILES vs SELFIES), pretraining dataset quality/size, policy architecture, and RL components (hill-climb replay buffer, KL regularization, log-likelihood penalties) for REINFORCE-based molecule generation with a pretrained GPT-style transformer, and combine the best choices into ChemRLformer.
Key results
- SMILES-based policies consistently outperform SELFIES-based policies across PyTDC and docking tasks.
- Pretraining on higher-quality ChEMBL data outperforms agents pretrained on 100x larger ZINC 100M data.
- Hill-climb replay buffer gives about 13% improvement in top-100 rewards, while KL regularization gives little benefit and Log P regularization boosts performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on aligned ChEMBL data rather than larger, less-aligned datasets | ChemRLformer · PyTDC and augmented docking tasks | reward | Helped |
| Use SMILES instead of SELFIES (ablation) | ChemRLformer · PyTDC tasks | reward | Helped |
| Use SMILES instead of SELFIES (ablation) | ChemRLformer · docking tasks | top-K reward | Helped |
| Use SELFIES instead of SMILES (ablation) | ChemRLformer · top-100 molecule set | diversity | Helped |
| Use SELFIES instead of SMILES (ablation) | ChemRLformer · molecule exploration | redundancy | Hurt |
| Use a transformer rather than an RNN (ablation) | ChemRLformer · molecular design tasks | reward | No clear effect |
| Use a fully connected architecture rather than a string-oriented architecture (ablation) | ChemRLformer · PyTDC tasks | reward | Hurt |
| Use a fully connected architecture rather than a string-oriented architecture (ablation) | ChemRLformer · docking tasks | docking reward | Mixed |
| Add a hill-climb replay buffer (ablation) | ChemRLformer RNN · PyTDC tasks | top-100 reward: 13% | Helped |
| Use a standard replay buffer (ablation) | ChemRLformer RNN · PyTDC tasks | reward | No clear effect |
| Add KL regularization toward the pretrained policy (ablation) | ChemRLformer RNN · PyTDC tasks | reward | No clear effect |
| Replace entropy regularization with Log p regularization (ablation) | ChemRLformer RNN · PyTDC tasks | reward | Helped |
Surprising findings
- FC policies exploit the docking reward by generating long chains of carbon and nitrogen atoms, obtaining unusually high docking rewards.
- Dataset quality matters more than dataset size for pretraining the RL policy.
Limitations
- Docking-based rewards are exploitable, so augmented docking was needed to avoid chemically trivial molecules.
- Results limited to a small set of architectures (FC, GRU, GPT-style transformer) of small size; scaling model size not studied.
Experimental setup
Datasets: ZINC 250K ZINC 1M ZINC 10M ZINC 100M ChEMBL
Benchmarks: PyTDC oracle tasks (21-22) docking tasks (fa7, parp1, 5ht1b, jak2, braf) augmented docking tasks
Metrics: top-1/top-10/top-100 average score diversity redundancy
Baselines: REINVENT REINVENT 2.0 SMILES-LSTM BOSS STONED Pasithea SMILES-VAE PPO
Builds on: REINFORCE REINVENT GuacaMol benchmark Therapeutics Data Commons
Closest papers by meaning
- MANDREL: Modular Reinforcement Learning Pipelines for Material Discovery (AAAI 2024 · similarity 0.89)
- Aligning Transformers with Continuous Feedback via Energy Rank Alignment (NeurIPS 2025 · similarity 0.89)
- Efficient Evolutionary Search Over Chemical Space with Large Language Models (ICLR 2025 · similarity 0.88)
- BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Protein Design with Agent Rosetta: A Case Study for Specialized Scientific Agents (ICML 2026 · similarity 0.87)
- Multi-Objective Molecular Design Through Learning Latent Pareto Set (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.