Research map / Architectures and efficiency / Attention and transformers
LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LMUFormer, a sequential Legendre Memory Unit model with convolutional patch embedding and channel mixer, plus a spiking variant, achieving strong Speech Commands V2 accuracy at far lower parameter and FLOP cost than transformers.
Problem
Transformers have high complexity and cannot process data sequentially, making them ill-suited for resource-constrained streaming edge applications; prior RNN/recurrent-transformer reformulations lose significant accuracy.
Approach
Augment the LMU memory cell with convolutional patch embedding (kernel size 3, causal-style small look-ahead) and a convolutional channel mixer so the model can train in parallel and run sequentially, then convert it into a spiking version where the memory state and channel mixing use LIF neurons.
Key results
- On Speech Commands V2 (35 classes), spiking LMUFormer reaches 96.12% accuracy, a new SNN SOTA among compared low-complexity SNNs.
- LMUFormer (non-spiking) reaches 96.53% vs AST 97.74%-level transformer accuracy with about 53x fewer parameters (1.622M vs 86.933M) and about 65x fewer FLOPs (1.89e8 vs 1.24e10).
- Spiking LMUFormer keeps 99% of its accuracy (95.17% vs 96.12%) while using 32.03% shorter sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add states to the patch embedding and channel mixer | LMUFormer · Speech Commands V2 | accuracy: 96.53 -> 96.92 | Helped |
| Replace the LMU module with the spiking LMU variant | LMUFormer · LRA average | accuracy: 61.08 -> 62.24 | Helped |
| Use 87 of 128 sequence samples | Spiking LMUFormer · Speech Commands V2 test | accuracy: 96.12 -> 95.17 | Hurt |
| Add a one-layer convolutional patch embedding to the naive LMU (ablation) | Non-spiking LMUFormer · Speech Commands V2 | accuracy: 76.17 -> 89.28 | Helped |
| Add a convolution and batch normalization layer after the LMU cell (ablation) | Non-spiking LMUFormer · Speech Commands V2 | accuracy: 89.28 -> 89.8 | Helped |
| Replace one-layer convolutional patch embedding with Conv PE (ablation) | Non-spiking LMUFormer · Speech Commands V2 | accuracy: 89.8 -> 95.57 | Helped |
| Add the convolutional channel mixer (ablation) | Non-spiking LMUFormer · Speech Commands V2 | accuracy: 95.57 -> 96.53 | Helped |
| Add a convolution and batch normalization layer after the spiking LMU cell (ablation) | Spiking LMUFormer · Speech Commands V2 | accuracy: 83.78 -> 87.88 | Helped |
| Replace one-layer convolutional patch embedding with Conv PE (ablation) | Spiking LMUFormer · Speech Commands V2 | accuracy: 87.88 -> 95.20 | Helped |
| Add the convolutional channel mixer (ablation) | Spiking LMUFormer · Speech Commands V2 | accuracy: 95.20 -> 96.12 | Helped |
| Use the spiking variant instead of the non-spiking model | LMUFormer · Speech Commands V2 | accuracy: 96.53 -> 96.12 | Hurt |
Surprising findings
- Spiking LMUFormer outperforms its non-spiking counterpart on the LRA benchmark by about 0.53% on average.
Limitations
- Still about 1.5% below the best transformer (AST) on Speech Commands V2 for the non-spiking model.
- Substantial gap to S4 on Long Range Arena; no pre-training capability as transformers have.
- Lacks large-scale streaming datasets for pre-training.
Experimental setup
Datasets: Speech Commands V2 permuted sequential MNIST Long Range Arena
Benchmarks: Speech Commands V2 permuted sequential MNIST Long Range Arena
Metrics: accuracy parameters FLOPs SOPs sequence length reduction
Baselines: AST KWT2 Res15 RadLIF SFA LIF Spikformer Spike-driven ViT S4 Nystromformer Linformer BigBird
Builds on: Legendre Memory Unit MLP-Mixer Spikformer Spike-driven ViT Long Range Arena
Closest papers by meaning
- Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity (ICML 2025 · similarity 0.88)
- Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model (ICML 2025 · similarity 0.87)
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.87)
- IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs (ICLR 2024 · similarity 0.87)
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.86)
- Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.