Research map / Reinforcement learning / Visual and representation RL
Improved Off-policy Reinforcement Learning in Biological Sequence Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes δ-Conservative Search, which masks tokens of high-score offline sequences with probability δ (adaptive via proxy uncertainty) and denoises them with a GFlowNet policy, to make off-policy GFlowNet active learning robust to proxy misspecification in biological sequence design.
Problem
Active learning for biological sequence design with RL/GFlowNets suffers from proxy misspecification on out-of-distribution sequences, which leads to unreliable rewards and poor high-score discovery under limited oracle budgets.
Approach
Conservative off-policy search: start from high-score offline sequences, inject token-level noise with Bernoulli(δ) masking, and let the GFlowNet forward policy denoise them, with δ set per data point from proxy ensemble/MC-dropout uncertainty so that uncertain regions get more conservative search.
Key results
- On RNA-A, GFN-AL + δ-CS reaches max 1.055 and mean 0.947 after ten rounds vs. 1.030 and 0.849 for GFN-AL, outperforming all listed baselines.
- On AAV (1,000 initial points), GFN-AL + δ-CS reaches max 0.704 vs. 0.560 for GFN-AL.
- On GFP (FLEXS), GFN-AL + δ-CS achieves mean top-128 of 3.569 vs. 3.508 for GFN-AL, with max 3.592 above the dataset max of 3.572.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add δ-Conservative Search to GFN-AL | GFN-AL · RNA-A | Median: 0.838 ± 0.013 -> 0.939 ± 0.008 | Helped |
| Add δ-Conservative Search to GFN-AL | GFN-AL · RNA-B | Median: 0.858 ± 0.004 -> 0.929 ± 0.004 | Helped |
| Add δ-Conservative Search to GFN-AL | GFN-AL · RNA-C | Median: 0.774 ± 0.004 -> 0.972 ± 0.043 | Helped |
| Add δ-Conservative Search to GFN-AL | GFN-AL · TF-Bind-8 | Median: 0.947 ± 0.004 -> 0.971 ± 0.006 | Helped |
| Add δ-Conservative Search to GFN-AL | GFN-AL · GFP | Mean: 3.508 ± 0.004 -> 3.569 ± 0.003 | Helped |
| Add δ-Conservative Search to GFN-AL (UCB) | GFN-AL (UCB) · AMP | Mean: 0.919 -> 0.938 | Helped |
| Add δ-Conservative Search to GFN-AL (EI) | GFN-AL (EI) · AMP | Mean: 0.940 -> 0.958 | Helped |
| Use δ=0.01 instead of δ=1 (ablation) | GFN-AL · GFP Medium | Median top-128 score: 0.21 -> 1.06 | Helped |
| Use δ=0.01 instead of δ=1 (ablation) | GFN-AL · GFP Hard | Median top-128 score: 0.09 -> 0.86 | Helped |
| Add δ-Conservative Search | SQL · GFP | Mean score of Top-128 after ten active rounds: 3.331 -> 3.573 | Helped |
| Add δ-Conservative Search | SQL · AAV | Mean score of Top-128 after ten active rounds: 0.480 -> 0.495 | Helped |
| Add δ-Conservative Search | VarGrad · GFP | Mean score of Top-128 after ten active rounds: 3.331 -> 3.567 | Helped |
Surprising findings
- On GFP, GFN-AL + δ-CS has lower diversity (46.3) than GFN-AL (60.3), trading diversity for reward.
- On the AAV task, GFN-AL + δ-CS scores 0.480 on Top-128 for SQL-like variants without δ-CS gains in some cases; gains are not uniform across soft RL variants (e.g., SQL on AAV 0.480 → 0.495).
Limitations
- Does not fundamentally resolve active learning drawbacks; relies on proxy quality and uncertainty estimates.
- Gains depend on tuning δ_const and λ per task; the paper notes domain-specific tuning can yield further gains.
- Lower diversity than GFN-AL on some protein tasks.
Experimental setup
Datasets: TF-Bind-8 RNA binding tasks (ViennaRNA) GFP (FLEXS) AAV (FLEXS) AMP
Benchmarks: Hard TF-Bind-8 FLEXS RNA-A/B/C TF-Bind-8 GFP AAV AMP GFP medium/hard (LatProtRL setup)
Metrics: max score median score mean score of top-K diversity novelty
Baselines: AdaLead BO TuRBO CMA-ES CbAS DbAS DyNA PPO GFN-AL GFNSeqEditor LS-GFN COMs LatProtRL
Builds on: GFlowNet GFN-AL trajectory balance LS-GFN FLEXS
Closest papers by meaning
- Designing Biological Sequences without Prior Knowledge Using Evolutionary Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Offline Model-Based Optimization via Policy-Guided Gradient Search (AAAI 2024 · similarity 0.86)
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.86)
- Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design (ICLR 2025 · similarity 0.86)
- ProSpero: Active Learning for Robust Protein Design Beyond Wild-Type Neighborhoods (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.