Research map / Architectures and efficiency / RNNs and normalization
pLSTM: parallelizable Linear Source Transition Mark networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces pLSTM, a multi-dimensional linear RNN that generalizes xLSTM/mLSTM gating (Source, Transition, Mark) to general DAGs and 2D grids, with a logarithmic-time parallel scan, P- and D-modes for long-range stability, and evaluates on arrow-pointing extrapolation, ImageNet-1K, and TUDataset graphs.
Problem
Modern linear RNNs (Mamba, xLSTM, GLA) are restricted to sequences or to pre-defined scan orders over multi-dimensional data, causing poor long-range propagation in images and graphs; multi-dimensional RNNs are non-parallelizable and lack associative memory.
Approach
Extend linear RNNs to act on the line graph of a general DAG using Source, Transition, and Mark gates on edges; this yields a hierarchical associative-scan parallelization in multiple dimensions and two stability modes (P-mode directional propagation with L1-bounded transitions, D-mode with multitree line graph) that are alternated across layers.
Key results
- On arrow pointing extrapolation to 384x384, pLSTM reaches 0.778 ± 0.031 test accuracy (extrapolation) versus much lower results for ViL and Mamba2D, which do not extrapolate beyond random.
- On ImageNet-1K ablation, pLSTM achieves 75.51 top-1 versus 73.49 for a re-trained DeiT-style ViT-T.
- On TUDataset, pLSTM gets the best MUTAG accuracy (0.8512) and an average of 0.7368, competitive with GIN (0.7468 avg).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pLSTM instead of ViT | ImageNet-1k, ViT-T scale · ImageNet-1k | top-1 accuracy: 73.49 -> 75.51 | Helped |
| Remove positional embeddings (ablation) | pLSTM · ImageNet-1k, ViT-T scale | top-1 accuracy: 75.51 -> 75.22 | Hurt |
| Use P-mode only instead of alternating P- and D-modes (ablation) | pLSTM · ImageNet-1k, ViT-T scale | top-1 accuracy: 75.51 -> 74.86 | Hurt |
| Use D-mode only instead of alternating P- and D-modes (ablation) | pLSTM · ImageNet-1k, ViT-T scale | top-1 accuracy: 75.51 -> 75.13 | Hurt |
| Use STM bias-only gates (ablation) | pLSTM · ImageNet-1k, ViT-T scale | top-1 accuracy: 75.51 -> 75.10 | Hurt |
| Remove positional embeddings (ablation) | pLSTM · Arrow Pointing | accuracy: 0.972 -> 0.975; 0.778 -> 0.769 | Mixed |
| Use P-mode only instead of alternating P- and D-modes (ablation) | pLSTM · Arrow Pointing | accuracy: 0.972 -> 0.978; 0.778 -> 0.746 | Mixed |
| Use D-mode only instead of alternating P- and D-modes (ablation) | pLSTM · Arrow Pointing | accuracy: 0.972 -> 0.957; 0.778 -> 0.828 | Mixed |
| Use pLSTM instead of the strongest listed graph baseline | TUDataset models · MUTAG | test accuracy: 0.8251 ± 0.10 -> 0.8512 ± 0.06 | Helped |
| Use pLSTM instead of the strongest listed graph baseline | TUDataset models · NCI1 | test accuracy: 0.8175 ± 0.02 -> 0.7324 ± 0.03 | Hurt |
| Use pLSTM instead of the strongest listed graph baseline | TUDataset models · PROTEINS | test accuracy: 0.7503 ± 0.04 -> 0.7502 ± 0.05 | No clear effect |
| Use pLSTM instead of the strongest listed graph baseline | TUDataset models · PTC_FM | test accuracy: 0.6162 ± 0.04 -> 0.6133 ± 0.08 | Hurt |
Surprising findings
- The D-mode-only ablation extrapolates better (0.828) than the full pLSTM (0.778) on arrow pointing, while being weaker at interpolation (0.957 vs 0.972).
Limitations
- Gap remains to highly optimized domain-specific models on ImageNet-1K (e.g., pLSTM-Vis-T 75.2 vs ViL-T 78.3 in the main table).
- Extrapolation is not perfect and the arrow-pointing task only tests long-range directional reasoning in a restricted way.
- Graph experiments show only comparable, not clearly superior, performance to GNN baselines.
Experimental setup
Datasets: ImageNet-1K TUDataset (MUTAG, NCI1, PROTEINS, PTC_FM) Arrow pointing synthetic dataset (100k at 192x192, validation at 192 and 384)
Benchmarks: Arrow Pointing Extrapolation ImageNet-1K TUDataset
Metrics: top-1 accuracy test accuracy extrapolation accuracy
Baselines: ViT EfficientNet ViL Mamba2D 2DMamba Vim GAT GCN GIN MPNN LSTM GNN DeiT-III
Builds on: xLSTM multi-dimensional LSTM DAG LSTM ViT DeiT-III
Closest papers by meaning
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.84)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.84)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.83)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.83)
- Geometric Generative Modeling with Noise-Conditioned Graph Networks (ICML 2025 · similarity 0.83)
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.