Research map / Time series / Causal and generative time series
Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SLiCEs, a framework of structured input-dependent linear CDE state-transition matrices (block-diagonal, sparse, Walsh-Hadamard, DPLR, diagonal) that keep maximal probabilistic expressivity while reducing cost, and evaluates them on A5 state tracking, regular language tasks, and UEA time series.
Problem
Parallel-in-time sequence models (SSMs, linear RNNs) are expressively limited, especially diagonal ones that cannot state-track, while dense linear CDEs that are maximally expressive are computationally infeasible at scale.
Approach
Replace the dense state-transition matrix of a linear neural CDE with structured input-dependent matrices (block-diagonal, sparse, or Walsh-Hadamard times diagonal) and prove they retain maximal probabilistic expressivity while cutting parameters and cost; block-diagonal is the most practical, combining efficient parallel scans and a chunk-wise diagonal part.
Key results
- BD-SLiCE and DPLR, WH, sparse and dense SLiCEs solve A5 with a single layer, while Mamba and D-SLiCE need growing depth with sequence length.
- On UEA-MTSCA, BD-SLiCE averages 64.0% accuracy vs 64.3% for Log-NCDE while reducing time per 1k steps from 1321.7s to 68.1s (about 20x).
- On regular language tasks, D-DE-SLiCE and DPLR-SLiCE both reach 85.1% average validation accuracy, the joint highest among parallel models; BD-SLiCE reaches 83.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the Log-NCDE nonlinear vector field with a block-diagonal linear vector field | Log-NCDE · UEA-MTSCA (six datasets) | test accuracy: 64.3 -> 64.0 | No clear effect |
| Replace the Log-NCDE nonlinear vector field with a block-diagonal linear vector field | Log-NCDE · UEA-MTSCA (six datasets) | time per 1,000 training steps: 1321.7 -> 68.1 | Helped |
| Use a block-diagonal state-transition structure instead of diagonal state-transition matrices | SLiCE · A5 state-tracking benchmark | minimum stacked layers for validation accuracy greater than 90% | Helped |
| Use a block-diagonal state-transition structure instead of Walsh-Hadamard structure | SLiCE · Regular language tasks | average validation accuracy: 64.5 -> 83.8 | Helped |
Surprising findings
- Diagonal SLiCEs (like Mamba) are provably not maximally expressive, illustrated by failure on parity with a 2-dimensional diagonal system, whereas a dense 2x2 rotation solves it exactly.
- The WH-SLiCE underperforms other SLiCE structures on the regular language and A5 length tasks despite being maximally expressive in theory.
Limitations
- Scaling to multi-billion parameter models remains challenging; needs efficient GPU kernels for matrix exponentials and scans for many small systems.
- Expressivity is established only asymptotically in hidden dimension; finite-dimensional expressivity remains open.
- Sparse SLiCEs lack efficient implementations so no practical speed-up is observed; SLiCEs cannot be stacked with Log-SLiCE because they output sequences not paths.
Experimental setup
Datasets: A5 state-tracking benchmark formal language benchmark (Chomsky hierarchy regular tasks) UEA Multivariate Time-Series Classification Archive (six datasets)
Benchmarks: A5 state tracking A5 length generalisation formal language regular tasks UEA-MTSCA six datasets
Metrics: validation accuracy test accuracy minimum stacked layers training time per 1k steps GPU memory
Baselines: LSTM xLSTM DeltaNet Gated DeltaProduct RWKV-7 Transformer S4D Mamba Log-NCDE LRU S5 S6
Builds on: Log-NCDE DeltaNet S4D Mamba Neural Rough Differential Equations
Closest papers by meaning
- Universal Time Series Generation with Neural Controlled Differential Equations (NeurIPS 2026 · similarity 0.88)
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.86)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.86)
- Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models (NeurIPS 2025 · similarity 0.86)
- Parallelizing non-linear sequential models over the sequence length (ICLR 2024 · similarity 0.85)
- Sequence Modeling with Spectral Mean Flows (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.