Research map / Continual learning / Online and replay-based
Tackling Continual Offline RL through Selective Weights Activation on Aligned Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQ-CD, a continual offline RL method that uses vector-quantized encoders to align heterogeneous state/action spaces and a U-Net diffusion model with task-specific weight masks that are assembled after training.
Problem
Continual offline RL where tasks have different observation and action spaces, in addition to the conventional setting with identical spaces, while avoiding catastrophic forgetting.
Approach
Vector-quantize each task's states and actions into a shared codebook-based latent space, then train a single diffusion model with task-specific sparse weight masks (selective weights activation) and merge the task-related weights by summation after training.
Key results
- On Ant-dir task 10-15-19-25, VQ-CD reaches mean return 558.22 versus 478.19 for CoD and 402.53 for IL-rehearsal.
- Replacing vector quantization with AE or VAE alignment degrades performance on D4RL CL settings, with VAE worse than AE.
- DDIM with 10 steps (stride 20) gives a 19.76x generation speed-up over 200 steps (5.73s to 0.29s per generation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VQ-CD instead of Multitask CoD | Ant-dir continual offline RL · Ant-dir 10-15-19-25 | mean return: 485.15 -> 558.22 | Helped |
| replace vector quantization with an auto-encoder (ablation) | VQ-CD space alignment · D4RL continual learning | normalized score | Hurt |
| replace vector quantization with a variational auto-encoder (ablation) | VQ-CD space alignment · D4RL continual learning | normalized score | Hurt |
| use an MLP-based instead of a U-net-based diffusion model (ablation) | VQ-CD diffuser · D4RL continual learning | normalized score | Hurt |
| use a smaller state codebook (ablation) | VQ-CD QSA · D4RL state-space alignment | normalized score | Hurt |
| increase the state codebook size beyond 512 (ablation) | VQ-CD QSA · D4RL state-space alignment | QSA loss | No clear effect |
| increase action latent vectors from 5 to 7 (ablation) | VQ-CD QSA · D4RL action-space alignment | normalized score | No clear effect |
| use 10 DDIM diffusion steps instead of 200 | VQ-CD generation · Ant-dir 4-18-26-34-42-49 | generation speed-up ratio: 1× -> 19.76× | Helped |
| use a dense optimizer followed by weights assembling instead of a sparse optimizer | VQ-CD training · D4RL, Ant-dir, and CW continual learning | time consumption per update: 0.089 vs 0.198; 0.096 vs 0.197; 0.089 vs 0.195; 0.090 vs 0.206; 0.062 vs 0.239; 0.064 vs 0.214; 0.061 vs 0.218 | Helped |
| apply the pre-trained QSA module with VQ-CD rather than using state and action padding | VQ-CD · D4RL continual learning | normalized score | Mixed |
Surprising findings
- Sparse optimizer training was much slower than dense Adam (e.g., 0.198s vs 0.089s per update on D4RL), motivating the post-hoc weights assembling instead of sparse training.
- Mask pruning by absolute weight thresholds can release some task masks while keeping over 90% of performance, but only gives finite capacity for new tasks.
Limitations
- Capacity for tasks beyond the predefined sequence is finite, since masks consume the available weights.
- Sparse-optimizer alternative is too slow, so the method relies on post-hoc weight assembling.
- Simpler alignment baselines (padding) are used for D4RL in the main results, and gains are smaller for similar-quality datasets.
Experimental setup
Datasets: D4RL MuJoCo Ant-dir Continual World CW10
Benchmarks: MuJoCo Ant-dir Continual World CW10 D4RL
Metrics: average episodic return average success rate normalized score
Baselines: CoD Multitask CoD IL-rehearsal CoD-LoRA Diffuser-w/o rehearsal CoD-RCR MTDIFF DD-w/o rehearsal CRIL DGR EWC PackNet
Builds on: Diffuser Continual Diffuser (CoD) VQ-VAE DDIM PackNet
Closest papers by meaning
- BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · similarity 0.87)
- Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.87)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.86)
- Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control (NeurIPS 2024 · similarity 0.86)
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.86)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.