Research map / Architectures and efficiency / Attention and transformers
Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Centaurus, a family of generalized SSM networks whose blocks are expressed as tensor networks so the optimal einsum contraction order can be chosen for efficient training, and mixes depthwise, bottleneck and full SSM blocks CNN-style for raw-audio keyword spotting, denoising and ASR.
Problem
Deep SSM networks use uniform depthwise-separable blocks, limiting channel connectivity design flexibility; SSM training cost depends strongly on operation order, and fully SSM-based competitive ASR without attention or recurrence is unexplored.
Approach
Express SSM blocks as tensor networks/einsum expressions so that depthwise, group, full and bottleneck connectivity can be designed like CNN blocks, and pick the optimal contraction order per block (including FFT placement) to speed up training; heterogeneous hybrid stacks outperform homogeneous ones.
Key results
- Hybrid Centaurus achieves 3.12 PESQ on VB-DMD with 0.51M parameters and 0.29G FLOPs/sec, versus FRCRN 3.21 at 6.9M and 76.2G FLOPs/sec.
- Fully SSM-based Centaurus with Mamba macro-block reaches 4.4 / 10.2 WER on LibriSpeech test-clean/other in online mode, competitive with streaming Conformer at 4.6 / 9.9.
- Optimal contraction order gives training speedups over the naive order, reported as sometimes orders of magnitude for SSM blocks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a heterogeneous mixture of full, bottleneck, and pointwise bottleneck SSM blocks instead of a homogeneous block type | Centaurus KWS network · SC35 testset | classification error | Helped |
| Use homogeneous depthwise-separable or bottleneck SSM blocks instead of the hybrid design (ablation) | Centaurus speech-denoising network · VB-DMD testset | PESQ | Hurt |
| Add causal depthwise Conv1d with kernel size 4 before every SSM block (ablation) | Centaurus hybrid denoising network · VB-DMD testset | PESQ: 3.12 -> 3.25 | Helped |
| Add an FFN module to every SSM block (ablation) | Centaurus ASR model · Librispeech test clean | WER: 6.0 -> 5.4 | Helped |
| Add a causal convolutional block to every SSM block (ablation) | Centaurus ASR model · Librispeech test clean | WER: 6.0 -> 4.8 | Helped |
| Wrap core SSM layers in the Mamba macro-architecture (ablation) | Centaurus ASR model · Librispeech test clean | WER: 6.0 -> 4.4 | Helped |
| Use homogeneous full SSM blocks rather than the hybrid design (ablation) | Centaurus speech-denoising network · VB-DMD testset | PESQ: 3.12 vs 3.04 | Hurt |
Surprising findings
- DWS and bottleneck homogeneous variants suffered severe training plateaus on speech denoising and could not generate meaningful audio, while the full-block variant trained.
- Adding S6 data-gating did not fix the denoising training plateau.
Limitations
- Data-dependent (S6/Mamba-style) SSMs are not fully explored because they restrict contraction order and need custom kernels.
- ASR is trained with a simple supervised/distillation pipeline without self-supervised pretraining and is compared against streaming baselines only with minimal optimization.
Experimental setup
Datasets: Google Speech Commands (SC35) VoiceBank + DEMAND LibriSpeech
Benchmarks: Google Speech Commands SC35 VoiceBank + DEMAND (PESQ) LibriSpeech test/dev-clean/other WER
Metrics: classification error PESQ word error rate parameters FLOPs
Baselines: S4D S5 S6 (Mamba) FRCRN DeepFilterNet3 SEMamba PercepNet DEMUCS Conformer Wav2Vec2 ContextNet Transformer
Builds on: S4D S5 Mamba opt_einsum HiPPO
Closest papers by meaning
- On Feature Learning in Structured State Space Models (NeurIPS 2024 · similarity 0.85)
- Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning (NeurIPS 2025 · similarity 0.85)
- Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity (ICML 2025 · similarity 0.85)
- Elastic Spectral State Space Models for Train-Once Budgeted Inference (NeurIPS 2026 · similarity 0.84)
- Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks (NeurIPS 2026 · similarity 0.84)
- SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.