Research map / Reinforcement learning / Visual and representation RL
Decision Mixer: Integrating Long-term and Local Dependencies via Dynamic Token Selection for Decision-Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Decision Mixer, a Decision Transformer variant that dynamically selects tokens per layer via a router and hypernetwork (plus an auxiliary predictor for autoregressive sampling) to balance local and long-term dependencies in offline RL.
Problem
Conditional sequence modeling for offline RL must balance local Markov features and long-term dependencies; fixed attention (DT) or fixed convolution (DC) each fail on some task types, and full attention is computationally expensive.
Approach
A plug-and-play dynamic token selection in alternating mixer layers: a router scores tokens and a hypernetwork sets a top-k count; only selected tokens go through attention and FFN, unselected tokens bypass via residual; an auxiliary token-level predictor enables selection during autoregressive sampling.
Key results
- DM averages 84.7 on D4RL Gym tasks versus 82.2 for DC and 76.2 for DT, and 86.7 on AntMaze versus 54.5 for DC.
- On Maze2D medium, DM scores 95.2 versus 38.2 for DC; on Adroit, DM averages 51.0 versus 38.9 for GDT.
- DM reduces FLOPs by 47.0% relative to DT (398.5G vs 752.5G) on hopper-medium with a 0.5% parameter increase; the auxiliary predictor ablation drops hopper-medium from 98.1 to 93.7 without it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic token selection and the auxiliary predictor (ablation) | Decision Mixer · hopper-medium | normalized score: 67.6 -> 98.1 | Helped |
| add the auxiliary predictor to dynamic token selection (ablation) | Decision Mixer · walker2d-medium-expert | normalized score: 70.9 -> 112.7 | Helped |
| add dynamic token selection to the auxiliary-predictor variant (ablation) | Decision Mixer · pen-cloned | normalized score: 105.4 -> 117.0 | Helped |
| add dynamic token selection and the auxiliary predictor (ablation) | Decision Mixer · maze2d-medium | normalized score: 8.2 -> 95.2 | Helped |
| add dynamic token selection and the auxiliary predictor (ablation) | Decision Mixer · antmaze-umaze | normalized score: 32.1 -> 100.0 | Helped |
| replace the alternating mixer and transformer layers with all mixer layers (ablation) | Decision Mixer · hopper-medium | FLOPs: 398.5 G -> 330.1G | Helped |
| replace the alternating mixer and transformer layers with all mixer layers (ablation) | Decision Mixer · hopper-medium | memory usage: 4474M -> 3004M | Helped |
| increase context length from K=20 to K=60 (ablation) | Decision Mixer · halfcheetah-medium | normalized score: 43.5 -> 43.8 | No clear effect |
| increase context length from K=20 to K=120 (ablation) | Decision Mixer · walker2d-medium-expert | normalized score: 112.7 -> 110.6 | Hurt |
Surprising findings
- Using only mixer layers (DM-ALL) causes significant training loss fluctuations, so alternating with standard transformer layers was adopted for stability.
- Fixing the token-selection count to 50% of sequence length (removing dynamic selection) causes a significant performance drop.
Limitations
- Robustness in incomplete or noisy data scenarios is not yet enhanced, as acknowledged by the authors.
- Performance is slightly below TD3+BC on some low-quality or low-coverage datasets.
Experimental setup
Datasets: D4RL Gym D4RL Adroit D4RL Kitchen D4RL Maze2D D4RL AntMaze Cheetah-vel Ant-dir MetaWorld ML10 MetaWorld ML45
Benchmarks: D4RL Gym D4RL Adroit D4RL Kitchen D4RL Maze2D D4RL AntMaze Cheetah-vel Ant-dir MetaWorld
Metrics: normalized score zero-shot return FLOPs memory parameter count
Baselines: Decision Transformer Decision ConvFormer CQL IQL TD3+BC StAR GDT CGDT LSDT DD
Builds on: Decision Transformer Decision ConvFormer Mixture of Experts Prompt-DT Random-LTD
Closest papers by meaning
- Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL (NeurIPS 2024 · similarity 0.87)
- Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling (NeurIPS 2024 · similarity 0.87)
- Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making (ICLR 2024 · similarity 0.87)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.86)
- Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.86)
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.