Research map / Architectures and efficiency / Attention and transformers
A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LRAM, a multi-task large action model with an xLSTM backbone trained by supervised sequence modeling on 432 RL tasks (894M transitions), showing comparable-or-better performance and faster inference than Transformer Decision Transformers.
Problem
Transformer-based large action models have high inference latency and memory cost, making them impractical for real-time robotics control with long contexts.
Approach
Replace the Transformer backbone of multi-task Decision-Transformer-style agents with modern recurrent architectures (xLSTM, Mamba) that train in parallel but infer with constant per-step cost, combined with unified image/state encoders, a shared discrete action head, and no action tokens in the context.
Key results
- At 206M parameters, both recurrent backbones (xLSTM and Mamba) outperform the Transformer (DT) in validation perplexity and normalized evaluation scores across all four model sizes (16M to 206M).
- xLSTM outperforms competitors on three of six domains at 206M, and tends to outperform Mamba at larger scales.
- In inference latency on A100 with Atari Freeway, DT runs out of memory at long context lengths while xLSTM and Mamba have constant per-step cost; DT goes OOM for batch sizes above 64 in throughput tests.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use xLSTM or Mamba instead of a Transformer backbone | Large action models · 432 training tasks across six domains | normalized score | Helped |
| Scale xLSTM to 206M parameters rather than using Mamba | Large action models · 432 training tasks across six domains | normalized score | Helped |
| Use a recurrent backbone instead of a Transformer | Large action models · Hold-out trajectories across six domains | perplexity | Helped |
| Use xLSTM [7:1] rather than other tested recurrent backbones | Algorithm Distillation · Dark-Room 10×10, 20 hold-out tasks | score | Helped |
| Use recurrent xLSTM or Mamba inference instead of Transformer inference | 206M action models · Atari Freeway, longer contexts and larger batches | inference latency | Helped |
| Use xLSTM or Mamba rather than DT | 206M action models · Atari Freeway, varying batch sizes at C=1600 | throughput | Helped |
| Remove previous actions from the context (ablation) | LRAM · Meta-World and DMControl | normalized score | Helped |
| Remove previous actions from the context (ablation) | LRAM · Discrete control environments | normalized score | No clear effect |
| Increase context length beyond 1 while retaining actions in training inputs (ablation) | LRAM · Meta-World and DMControl | normalized score | Hurt |
| Increase context length when training without actions (ablation) | LRAM · Training domains | normalized score | Helped |
| Exclude return-to-go and reward tokens (ablation) | 206M LRAM · Training-task evaluation | normalized score | No clear effect |
| Use an xLSTM mLSTM:sLSTM ratio of [3:1] or place sLSTM blocks in lower layers (ablation) | xLSTM LRAM · 432 tasks and Dark-Room | score | Mixed |
Surprising findings
- Removing action tokens from the context improves performance on continuous-control domains (Meta-World, DMControl), attributed to alleviating the copycat problem.
- Longer context lengths hurt performance on Meta-World and DMControl when actions are included, the reverse of what holds when actions are removed.
Limitations
- No experiments on real robots; all evaluation is in simulation.
- Fine-tuning experiments limited to offline RL; ICL evaluated only in a grid-world (Dark-Room).
- Inference speed gains depend on device, model size, batch size and context length; Mamba slower at B=1 due to torch.compile incompatibility.
Experimental setup
Datasets: DQN-Replay (Atari) Composuite datasets Meta-World datasets (MT45) DMControl (generated with SAC agents) Mimicgen datasets Procgen datasets
Benchmarks: Atari Meta-World DMControl Composuite Mimicgen Procgen Dark-Room 10x10
Metrics: validation perplexity human-normalized score data-normalized score latency throughput memory consumption
Baselines: Decision Transformer Mamba xLSTM [1:0]
Builds on: Decision Transformer multi-game Decision Transformer xLSTM Mamba Algorithm Distillation
Closest papers by meaning
- xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference (ICML 2025 · similarity 0.88)
- Latent Action Reparameterization for Efficient Agent Inference (NeurIPS 2026 · similarity 0.87)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling (NeurIPS 2024 · similarity 0.86)
- Recurrent Reinforcement Learning with Memoroids (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.