Research map / Architectures and efficiency / Attention and transformers

Elastic Spectral State Space Models for Train-Once Budgeted Inference

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ES-SSM, a train-once spectral SSM whose Hankel spectral-channel prefixes can be truncated into compact models, trained with input-adaptive gates and budget dropout.

Problem

Sequence models are trained at a fixed compute capacity, yet deployment needs multiple inference budgets; naive truncation of spectral SSM channels degrades sharply because learned information is spread across channels.

Approach

Elasticity via prefix truncation along fixed Hankel spectral channels of the SSM convolution operator, made reliable by input-adaptive channel-wise sigmoid gates and budget dropout (log-scale sampled prefix budgets plus periodic full-budget anchors) so low-index channels become predictive alone.

Key results

• On SlimPajama, ES-SSM exported budgets from one 1.3B-scale checkpoint give smooth BPB curves, avoiding the severe low-budget collapse seen in MatFormer.
• On D4RL, ES-SSM's five exported budgets from one training run reduce total training wall-clock from 107.84 h to 23.99 h (77.8% saving).
• On Speech Commands V2, ES-SSM at full capacity is within 0.27 percentage points of Conformer, and K=3 retains almost all accuracy.

Surprising findings

• ES-SSM at several D4RL budgets matches or exceeds independently trained Transformers, with the largest gain at full capacity.

Limitations

• Evaluated only up to around 1.3B parameters and 30B byte-tokens; larger scales untested.
• Fixed set of runtime budgets; no online budget selection.
• Lacks mature systems-level efficiency optimization compared with attention kernels.

Experimental setup

Datasets: SlimPajama Long Range Arena Speech Commands V2 D4RL MuJoCo

Benchmarks: SlimPajama Long Range Arena Speech Commands V2 D4RL MuJoCo

Metrics: bits-per-byte accuracy normalized return latency throughput peak memory

Baselines: MatFormer MatMamba Transformer++ Pythia-style GLA Mamba-2 Conformer Transformer

Builds on: Hankel spectral filtering Matryoshka Representation Learning MatFormer MatMamba Once-for-All

Closest papers by meaning

• HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.87)
• Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.87)
• MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models (NeurIPS 2026 · similarity 0.86)
• Quamba: A Post-Training Quantization Recipe for Selective State Space Models (ICLR 2025 · similarity 0.86)
• Fractional State Space Transition for Long Sequence Modeling (NeurIPS 2026 · similarity 0.86)
• Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control (ICML 2026 · similarity 0.86)