Research map / Continual learning / Online and replay-based
Permanent and Transient Representations for Continual Reinforcement Learning
Nishanth Anand, Doina Precup
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a continual RL method that pairs permanent and transient representations as complementary learning systems to address the stability-plasticity trade-off, extending Anand & Precup (2023), which used permanent and transient value functions.
Problem
Continual reinforcement learning agents must adapt to new situations while retaining prior knowledge, facing a stability-plasticity trade-off.
Approach
Applies the complementary learning systems idea at the representation level, using separate permanent (slow, long-term) and transient (fast, adaptive) representations rather than only permanent and transient value functions as in Anand & Precup (2023).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use separate (first-order permanent, higher-order transient) features instead of all features in both PT-TD components (ablation) | PT-TD · 2D grid prediction (linear Fourier features) | RMSVE | Improved |
| restrict permanent features to first-order only in PT-TD (separate) vs using both orders (ablation) | PT-TD · 1D chain prediction | RMSVE after task change | Improved |
| use only one feature order in plain TD learning (ablation) | TD learning · 1D chain prediction | RMSVE | Hurt |
| use a single reward-correlated feature for the transient Q function (separate variant) (ablation) | PTQ-learning · 7-state chain control | adaptation speed: re-adjusts within five episodes after task change | Improved |
| reset weights at the start of each task (reset Q-learning) vs continual | Q-learning · 7-state chain control | episodic return | Hurt |
| add EWC or continual backprop to PQN | PQN · CIFAR image-task MDP | return | No clear effect |
| use fully neural-network transient component (PT-NN) instead of non-parametric transient (ablation) | PT · CIFAR image-task MDP | return | Hurt |
| use only the non-parametric transient approximator (T-Only), no permanent network (ablation) | PT · CIFAR image-task MDP | return | Hurt |
| pair neural permanent value function with MinHash non-parametric transient table | PQN · CIFAR image-task MDP (hidden task boundaries) | adaptation after task change | Improved |
| pair PQN with MinHash transient table | PQN(32) · Craftax-Classic 250M | steps to return 15: over 200M -> 150M steps | Improved |
| use MinHash instead of random hashing in the transient table (ablation) | transient table · symbolic observations | value correction quality | Improved |
Experimental setup
Builds on: Anand & Precup (2023) permanent and transient value functions
Closest papers by meaning
- Prediction and Control in Continual Reinforcement Learning (NeurIPS 2023 · similarity 0.92)
- Continual Knowledge Adaptation for Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.86)
- Knowledge Retention in Continual Model-Based Reinforcement Learning (ICML 2025 · similarity 0.86)
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
