Research map / Reinforcement learning / Visual and representation RL
Reinforcement Learning for Control of Non-Markovian Cellular Population Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a non-Markovian fractional-derivative model of phenotypic switching for drug dosing and applies deep RL (Double DQN, and noisy FQF for generalization across memory strengths) to learn bang-bang dosing policies that outperform memoryless and threshold baselines.
Problem
Optimal drug dosing to drive cell populations with phenotypic plasticity and memory (non-Markovian dynamics) toward extinction, where model parameters are unknown and exact optimal control is intractable.
Approach
Combines a bang-bang optimal control reduction (binary action space justified by Pontryagin's principle) with model-free deep RL on a fractional-differential-equation switching model, using frame-stacked growth-rate observations, and extends to a single generalist agent trained with domain randomization over memory strengths.
Key results
- In the memoryless case (mu=1), DQN recovers the known optimal policy: an initial drug phase followed by regular pulsing.
- For memory cases, the learned DQN policy outperforms memoryless and threshold-based baselines at a bounded simulation step Delta=0.01 h.
- PPO and SAC underperform DQN, failing to find stable bang-bang protocols; the generalist FQF agent adapts to memory strengths drawn from [0.6,1] and stays robust under observation noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the learned DQN dosing policy instead of the memoryless or resistant-fraction threshold protocols | Cell-population control in memory-based dynamics (μ<1) · Fixed memory environments | population control | Helped |
| use continuous-action PPO or SAC instead of discrete-action DQN | Cell-population control at μ=0.9 · Fixed memory μ=0.9 | population control | Hurt |
| train one agent across a range of memory strengths using domain randomization | FQF generalist agent · Variable memory strength μ∈[0.6,1] | population control across memory strengths | Helped |
| add observation noise to the agent's state | Learned RL policy · Memoryless environment with observation noise | population reduction | No clear effect |
| vary memory strength during an episode | Learned RL policy · Memory strength reset every 20 decision steps, drawn from [0.6,1] | population control | No clear effect |
| increase measurement frequency by reducing Δ beyond a threshold (ablation) | DQN policy · Fixed memory environments | population control performance | No clear effect |
| sweep the tested hyperparameters (ablation) | Double DQN and FQF agents · Fixed memory environments | agent performance | No clear effect |
| use only two successive frames to recover the policy in the memoryless case (ablation) | DQN · Memoryless dynamics (μ=1) | recovery of the optimal dosing policy | Helped |
Surprising findings
- Continuous-action algorithms (PPO, SAC) fail to discover bang-bang control and default to average actions.
- Model-free RL can control populations using only the growth rate observation, without access to the resistant fraction or model parameters.
Limitations
- Deterministic model only; stochastic population heterogeneity not considered.
- Frame-stacking rather than recurrent policies used to capture history.
- Clinical relevance untested; no real-world data; ignores patient well-being and drug penalties.
Experimental setup
Datasets: simulated phenotypic switching model with fractional memory kernel
Benchmarks: simulated cell population dynamics at fixed memory strengths generalization over memory strengths in [0.6,1] observation noise robustness
Metrics: log population change (cost) resistant fraction
Baselines: constant drug application memoryless optimal pulsing protocol resistant-fraction threshold pulsing PPO SAC
Builds on: DQN Double DQN FQF NoisyNets Stable-Baselines3
Closest papers by meaning
- The GATTACA Framework: Graph Neural Network-Based Reinforcement Learning for Controlling Biological Networks (AAAI 2026 · similarity 0.86)
- Adding Conditional Control to Diffusion Models with Reinforcement Learning (ICLR 2025 · similarity 0.84)
- Neuroplastic Expansion in Deep Reinforcement Learning (ICLR 2025 · similarity 0.83)
- Online Control in Population Dynamics (NeurIPS 2024 · similarity 0.83)
- When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL (NeurIPS 2024 · similarity 0.83)
- Feedback Efficient Online Fine-Tuning of Diffusion Models (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.