Research map / Architectures and efficiency / Attention and transformers

Fractional State Space Transition for Long Sequence Modeling

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FRAC, a selective SSM whose memory kernel comes from a Caputo fractional relaxation approximated by a log-spaced sum-of-exponentials bank, giving power-law memory with bounded-state recurrence and parallel scan.

Problem

Exponential forgetting in ODE-based SSMs limits retention over broad temporal ranges; fractional dynamics give heavy-tailed memory but are non-Markovian and incompatible with finite-state recurrent layers.

Approach

Approximate the Mittag-Leffler/fractional relaxation kernel by a finite, geometrically spaced bank of exponential modes (sum-of-exponentials), discretize with ZOH, and make it selective via token-dependent alpha, lambda, Delta and learned softmax read/write weights over modes with log-timescale priors.

Key results

• On LongBench, FRAC averages 17.9 vs 16.0 for GDN (about 1.9 points higher) and is best on 8/14 tasks.
• On heavy-tail synthetic probe at 128K, FRAC reaches 63.8% accuracy vs 58.4% for GDN, with best scores from 8K to 128K.
• On short-context LM Harness, FRAC averages 55.1 vs 55.3 for Transformer and Mamba3-MIMO, i.e. competitive.

Surprising findings

• On Recall-Retrieval, FRAC scores 20.0 on FDA versus 27.4 for Transformer and 28.9 for Mamba3-MIMO, a large gap on one task despite good average.
• Algorithm's beta uses 1-rho instead of the exact ZOH (1-rho)/lambda for training stability, so the implementation deviates from the derived discretization.

Limitations

• Theorem is existence-only; practical mode count M=16 chosen empirically rather than from bounds.
• Implementation deviates from exact ZOH injection (uses 1-rho instead of (1-rho)/lambda), and read/write priors are not uniquely determined by theory.
• Mixed results: weaker on some recall tasks (e.g., FDA) and on LongBench code tasks (LCC 11.0 vs 25.6 for Transformer).

Experimental setup

Datasets: FineWeb-Edu (SmolLM-Corpus) MADLab synthetic tasks HG38 human genome LongBench LM Harness tasks NIAH passkey tasks Recall-retrieval suite (SWDE, SQuAD-derived FDA, TQA, NQ, Drop)

Benchmarks: Heavy-tail synthetic probe MADLab Needle-in-a-haystack (S-NIAH variants, 1K-64K) LongBench LM Harness Recall-Retrieval HG38 DNA perplexity

Metrics: accuracy perplexity F1/score (LongBench)

Baselines: Mamba2 GDN Mamba3-SISO Mamba3-MIMO Transformer (Qwen3 architecture) RetNet FADE

Builds on: Mamba Mamba2 Gated DeltaNet Mamba-3 HyenaDNA

Closest papers by meaning

• LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning (NeurIPS 2026 · similarity 0.88)
• Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing (ICLR 2025 · similarity 0.86)
• Elastic Spectral State Space Models for Train-Once Budgeted Inference (NeurIPS 2026 · similarity 0.86)
• Longhorn: State Space Models are Amortized Online Learners (ICLR 2025 · similarity 0.86)
• Bridging Expressivity and Scalability with Adaptive Unitary SSMs (NeurIPS 2025 · similarity 0.85)
• MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models (NeurIPS 2026 · similarity 0.85)