Research map / Probabilistic and scientific modeling / Dynamical systems
Continuity Laws for Sequential Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes continuity (convergence under temporal refinement) as an inductive bias for sequential models, showing S4 is stable under refinement while S6 is not, a dataset continuity metric that predicts when continuous models win, and a stage-wise time-subsampling training method that speeds up S4D on LRA.
Problem
Whether SSMs motivated by continuous-time formulations actually behave continuously under temporal refinement, whether task continuity can be measured, and whether matching model and task continuity improves performance.
Approach
Define model continuity via first-order convergence of discrete outputs under temporal refinement (with a Lipschitz-based error bound per model), define task continuity via a lag-based local similarity metric relative to far-apart pairs, and exploit time-axis continuity by training S4D with progressively refined subsampled inputs.
Key results
- S4 relative error stays stable under input scaling while S6 error grows rapidly; S4 test error stable across refinement and discretization switches while S6 degrades.
- On the controlled continuity sweep, S4 underperforms S6 and Transformers on discrete inputs but outperforms both as continuity increases.
- Stage-wise subsampled S4D reaches target accuracies in as little as ~0.1-0.2x the vanilla time on Retrieval and PathX, and is faster on five of six LRA tasks.
Surprising findings
- Continuous-time motivation alone does not guarantee continuous behavior after discretization: S6 is markedly discontinuous under refinement despite its SSM origin.
- Subsampled training yields a higher final accuracy on several continuous tasks (Pathfinder, PathX, Image) and does not help S6.
Limitations
- The task continuity metric is acknowledged as one of possibly many; no claim it is the best measure.
- Stage-wise subsampling does not reduce inference cost and does not modify the model.
- Theoretical bounds rely on Lipschitz and boundedness assumptions and the analysis covers a single-input scalar setting extended informally to higher dimensions.
Experimental setup
Datasets: Long-Range Arena (ListOps, Retrieval, Text, Image, Pathfinder, PathX) synthetic dynamical systems trajectories (damped harmonic oscillator, Van der Pol, Ornstein-Uhlenbeck, forced Duffing)
Benchmarks: Long-Range Arena synthetic dynamical system forecasting/parameter recovery
Metrics: relative L-infinity error test error normalized relative gap continuity score mu wall-clock time to target accuracy test accuracy
Baselines: S4 S6 Transformer Linear Attention DeltaNet Mamba-3 vanilla S4D
Builds on: S4 S5 S6 (Mamba) HiPPO Neural ODEs
Closest papers by meaning
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.86)
- Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing (ICLR 2025 · similarity 0.85)
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.85)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.85)
- Temporal-Difference Variational Continual Learning (NeurIPS 2025 · similarity 0.85)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.