Research map / Reinforcement learning / Visual and representation RL
Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Decision ConvFormer, a MetaFormer-based offline RL action predictor that replaces Decision Transformer's attention token mixer with static, depthwise causal convolution filters over RTG/state/action tokens.
Problem
Decision Transformer's attention is argued to be an inappropriate, overparameterized structure for capturing the local Markovian dependence in offline RL trajectories, and it is costly to train.
Approach
Replace the attention token mixer with input-independent 1D depthwise causal convolution filters, using separate filters for RTG, state, and action tokens (filter length 6), within a MetaFormer block; motivated by direct learning of banded attention matrices in DT.
Key results
- DC averages 82.2 on MuJoCo locomotion versus 76.4 for DT, and 81.8 on AntMaze versus 65.8 for DT.
- On Atari (8 games), DC achieves mean 86.2 and hybrid DC 94.7, versus 62.2 for DT.
- DC uses 95% fewer token-mixer parameters than DT on MuJoCo/AntMaze and trains about 75% faster on Atari.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace attention with local convolution filters | DT-style action prediction · AntMaze mean, offline | expert-normalized return: 65.8 -> 81.8 | Helped |
| replace attention with local convolution filters | DT-style action prediction · Atari mean, offline | gamer-normalized return: 62.2 -> 86.2 | Helped |
| add a final attention block after convolution blocks (ablation) | DC · Atari mean, offline | gamer-normalized return: 86.2 -> 94.7 | Helped |
| replace three modality-specific filters with one shared filter (ablation) | DC · MuJoCo and AntMaze | expert-normalized return: 88.3 -> 92.5; 77.6 -> 79.2; 89.6 -> 94.1; 78.0 -> 76.6; 81.6 -> 85.0; 78.1 -> 78.5 | Mixed |
| increase filter length from 3 to 6 (ablation) | DC, hopper-medium · hopper-medium | expert-normalized return: 83.5 -> 92.5 | Helped |
| increase context length from 8 to 20 (ablation) | DC, hopper-medium · hopper-medium | expert-normalized return: 92.5 -> 94.2 | Helped |
| increase filter length from 6 to 30 (ablation) | DC, hopper-medium, K=20 · hopper-medium | expert-normalized return: 94.2 -> 93.5 | Hurt |
| reduce context length from 20 to 8 (ablation) | DT · hopper and walker2d medium and medium-replay | expert-normalized return: 77.0 -> 74.8; 73.4 -> 72.5 | Hurt |
| reduce context length from 8 to 2 (ablation) | DT · hopper and walker2d medium and medium-replay | expert-normalized return: 74.8 -> 72.5; 72.5 -> 71.6 | Hurt |
| reduce context length from 20 to 8 (ablation) | DT · halfcheetah medium and medium-replay | expert-normalized return: 39.8 -> 39.3 | No clear effect |
| replace ReLU with GELU (ablation) | DT and DC · AntMaze | expert-normalized return: 66.2 -> 69.4; 58.0 -> 62.2; 71.0 -> 85.0; 63.6 -> 78.5 | Helped |
| replace ReLU with GELU (ablation) | DT and DC · MuJoCo | expert-normalized return | No clear effect |
Surprising findings
- Directly learning a banded attention matrix outperforms DT's parameterized attention, suggesting DT's full lower-triangular attention is an artifact of parameterization.
- Attention-free replacement of DT's token mixer with a single convolution filter still substantially beats DT in most settings.
Limitations
- Markov/local assumption may be weak for tasks requiring long-range credit assignment, addressed only by a hybrid attention layer.
- Hyperparameters such as target RTG and context length were tuned per benchmark, with best-of-six RTG reported.
Experimental setup
Datasets: D4RL MuJoCo D4RL AntMaze Atari DQN replay (1%)
Benchmarks: D4RL MuJoCo D4RL AntMaze Atari
Metrics: expert-normalized return gamer-normalized return training time GPU memory parameter count
Baselines: TD3+BC CQL IQL DT ODT RvS DS4 BC
Builds on: Decision Transformer MetaFormer ODT CQL
Closest papers by meaning
- Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.90)
- Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement (NeurIPS 2024 · similarity 0.89)
- In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought (ICML 2024 · similarity 0.89)
- Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling (NeurIPS 2024 · similarity 0.88)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.88)
- Decision SpikeFormer: Spike-Driven Transformer for Decision Making (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.