Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Deep Reinforcement Learning with Time-Scale Invariant Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Integrates a computational-neuroscience scale-invariant memory (Laplace-transform-based CogRNN with inverse Laplace time cells) into A2C/A3C-style deep RL agents, showing learning speed is roughly invariant across temporal scales unlike LSTM/RNN cores.
Problem
Deep RL agents with standard recurrent memories (LSTM, RNN) learn temporal relationships only at a limited set of time scales and need hyperparameter retuning when task timing is rescaled, unlike animals that show scale-invariant learning.
Approach
Replace the learned recurrent core with a fixed, analytically computed two-layer network that approximates a real-domain Laplace transform followed by an approximate inverse Laplace transform, producing log-compressed, scale-invariant time-cell memory so temporal rescaling becomes translation; combined with convolution and max-pooling, this yields rescaling-invariant policies.
Key results
- CogRNN agents reached high performance across all four rescaled tasks with similar learning speed at different temporal scales, while LSTM and RNN learning speed varied with scale and LSTM/RNN failed to learn the 3D interval timing task.
- In the 1D interval timing task trained at scale 1 with REINFORCE, CogRNN with convolution and pooling achieved perfect performance without retraining at 2x and 4x time scales, whereas RNN agents failed to generalize.
- Only CogRNN neurons showed the linear relationship between time-cell peak time and standard deviation expected for scale-invariant representations (Fig. S4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CogRNN instead of LSTM | Interval reproduction agents · Interval reproduction, training interval 60 | accuracy: .2±.1 -> 1.0±.0 | Helped |
| Use CogRNN instead of LSTM | Interval reproduction agents · Interval reproduction, validation interval 65 | accuracy: .1±.1 -> 1.0±.1 | Helped |
| Use CogRNN instead of LSTM | Interval reproduction agents · Interval reproduction, training interval 10 | accuracy: 1.0±.0 -> 1.0±.0 | No clear effect |
| Add the inverse Laplace transform and time-cell representation (ablation) | CogRNN · 3D interval timing | performance | Helped |
| Replace trainable recurrent weights with frozen RNN weights (ablation) | 3D interval timing agents · 3D interval timing | performance | Hurt |
| Replace trainable recurrent weights with frozen LSTM weights (ablation) | 3D interval timing agents · 3D interval timing | performance | Hurt |
| Vary tau_star_min (ablation) | CogRNN · 3D interval timing | performance | Mixed |
| Vary tau_star_max (ablation) | CogRNN · 3D interval timing | performance | Mixed |
| Vary k (ablation) | CogRNN · 3D interval timing | performance | Mixed |
| Vary the learning rate (ablation) | CogRNN and LSTM · 3D interval timing | performance | Mixed |
| Vary the entropy coefficient (ablation) | CogRNN and LSTM · 3D interval timing | performance | Mixed |
| Vary the horizon length (ablation) | CogRNN and LSTM · 3D interval timing | performance | Mixed |
Surprising findings
- Frozen-weight LSTM and RNN agents failed to learn the 3D interval timing task, and CogRNN without the inverse Laplace transform (F only) also performed poorly.
Limitations
- Scale invariance is imperfect due to edge effects from numerical derivatives and finite k_max range; only holds for temporal relationships within the tmin-tmax hyperparameter range.
- Results on the 3D environment rely on hyperparameter selection (learning rate, k, tau range) and the RL algorithm (A2C) still contains scale-breaking components such as exponential discounting.
Experimental setup
Datasets: custom 1D interval timing environment custom 3D PyBullet T-maze interval timing environment custom interval discrimination environment NeuroGym delayed-match-to-sample custom interval reproduction environment
Benchmarks: 3D interval timing 1D interval timing interval discrimination delayed-match-to-sample interval reproduction
Metrics: mean reward vs trials accuracy psychometric curves time-cell peak time vs standard deviation
Baselines: LSTM simple RNN CogRNN F (no inverse Laplace) frozen LSTM frozen RNN
Builds on: A3C generalized advantage estimation Weber-Fechner scale-invariant memory model (Shankar & Howard) REINFORCE
Closest papers by meaning
- Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments (NeurIPS 2025 · similarity 0.87)
- Balancing Context Length and Mixing Times for Reinforcement Learning at Scale (NeurIPS 2024 · similarity 0.86)
- Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning (ICML 2024 · similarity 0.86)
- Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning (ICLR 2025 · similarity 0.86)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.85)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.