Research map / Reinforcement learning / Offline RL
Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Decision Mamba (DM), a Mamba-based offline RL sequence model with a fine-grained intra-step SSM and a progressive self-evolution regularization that refines action targets with the policy's own past predictions.
Problem
Transformer-based conditional sequence models for offline RL handle out-of-distribution states/actions poorly, under-use historical inter-step information, ignore intra-step RTG-state-action structure, and overfit noisy suboptimal trajectories.
Approach
Replace the transformer with a Mamba selective SSM and add a parallel fine-grained SSM over each step's RTG-state-action triplet, fused with the coarse-grained SSM; regularize training by replacing action labels with a mix of the label and the policy's previous prediction, with a linearly growing weight beta.
Key results
- Average normalized score of 83.2 over nine Gym-MuJoCo medium/medium-expert/medium-replay datasets, vs 79.1 for the best baseline GDT.
- On AntMaze-umaze and umaze-diverse, DM scores 100.0 and 90.0 vs 80.0 and 70.0 for LaMo.
- Removing PSER drops the average from 83.2 to 77.2, the largest ablation drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Decision Mamba instead of the strongest reported baseline | Gym-MuJoCo medium, medium-expert, and medium-replay tasks · Gym-MuJoCo (Avg.) | normalized score: 79.1 -> 83.2 | Helped |
| Use Decision Mamba instead of LaMo | Extensive benchmark suite · Extensive Results (Avg.) | normalized score: 102.0 -> 107.9 | Helped |
| Add the multi-grained branch (ablation) | Decision Mamba · Gym-MuJoCo (Avg.) | normalized score: 79.2 -> 83.2 | Helped |
| Add progressive self-evolution regularization (ablation) | Decision Mamba · Gym-MuJoCo (Avg.) | normalized score: 77.2 -> 83.2 | Helped |
| Add inverse learning objectives (ablation) | Decision Mamba · Gym-MuJoCo (Avg.) | normalized score: 80.8 -> 83.2 | Helped |
| Set βK to 1 instead of using the default DM setting (ablation) | Decision Mamba PSER · Gym-MuJoCo (Avg.) | normalized score: 83.2 -> 80.9 | Hurt |
| Set βK to 0.75 instead of using the default DM setting (ablation) | Decision Mamba PSER · Gym-MuJoCo (Avg.) | normalized score: 83.2 -> 80.3 | Hurt |
| Set βK to 0.5 instead of using the default DM setting (ablation) | Decision Mamba PSER · Gym-MuJoCo (Avg.) | normalized score: 83.2 -> 81.4 | Hurt |
| Set βK to 0.25 instead of using the default DM setting (ablation) | Decision Mamba PSER · Gym-MuJoCo (Avg.) | normalized score: 83.2 -> 80.2 | Hurt |
| Set βK to 0, removing self-evolution capability (ablation) | Decision Mamba PSER · Gym-MuJoCo (Avg.) | normalized score: 83.2 -> 77.2 | Hurt |
Surprising findings
- Setting beta_K=0 (no self-evolution) gives the worst overall performance, while w/o ILO slightly improves Hopper-M (94.0 vs 98.5 is lower, but avg 80.8 is above w/o MG only in some cells).
Limitations
- Computational efficiency advantage is claimed but not measured.
- Gains on several datasets are small or negative relative to CQL and other baselines (e.g., HalfCheetah-M, Hopper-M-R vs TT/Diffuser).
Experimental setup
Datasets: D4RL Gym-MuJoCo (HalfCheetah, Hopper, Walker, Ant) D4RL AntMaze (umaze, umaze-diverse)
Benchmarks: D4RL Gym-MuJoCo D4RL AntMaze
Metrics: normalized score (mean and std over 4 seeds)
Baselines: BC CQL DT RvS StARformer GDT Waypoint Transformer EDT LaMo TT CGDT Diffuser
Builds on: Decision Transformer Mamba S4 LaMo Conservative Q-Learning
Closest papers by meaning
- Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling (NeurIPS 2024 · similarity 0.92)
- Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning? (NeurIPS 2024 · similarity 0.89)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.89)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.88)
- MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.