Research map / Architectures and efficiency / Attention and transformers

MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MaRK, which maps diffusion-timestep context to bounded modulations of a frozen Hydra-BERT SSM's A, B, C, D, Δ parameters via Hypernet, Chebyshev, or DCT adapters, for diffusion LM adaptation with 6.3–11M trainable parameters.

Problem

Conditioning pre-trained SSMs for iterative (diffusion) generation typically acts outside the recurrent operator via input injection or AdaLN, leaving the temporal dynamics fixed.

Approach

Map the context/timestep embedding directly to bounded, low-rank modulations of the SSM recurrence, read-in, read-out, skip and discretization parameters, so each diffusion step reshapes the model's Markov parameter (impulse-response) sequence; stability follows from the bounded log-space parameterization.

Key results

• Chebyshev adapter (11.0M params) achieves the lowest average CART validation loss of 2.55, vs DCT 2.59 (7.1M) and Hypernet 3.77 (6.3M) across five datasets, 10 seeds.
• Frozen-backbone ablation: modulating only A gives 3.44 on WikiText for Chebyshev vs 2.55 with full modulation and 4.00 with none.
• At sequence length 4096, MaRK adds 14–16% latency over frozen Hydra; memory overhead stays below 1%.

Surprising findings

• Hypernet adapter (most expressive) is far worse than the structured Chebyshev and DCT bases, by about 1.2 nats on average.
• Modulating Delta alone gives little benefit, and modulating only B,C (Mamba-style selection) falls well short of full modulation.

Limitations

• No controlled head-to-head comparison against AdaLN or input injection on reconstruction quality or downstream tasks.
• No scaling study beyond the 111M Hydra-BERT backbone; no generation quality evaluation.
• Stability and identifiability guarantees rely on stated assumptions and the synthetic recovery setup.

Experimental setup

Datasets: PTB WikiText Lambada AG News Arxiv

Benchmarks: PTB WikiText Lambada AG News Arxiv

Metrics: CART loss Markov operator error inference latency memory overhead

Baselines: Input Injection AdaLN Zero Frozen Hydra Hypernet DCT

Builds on: Hydra Mamba LPV systems Dream (token-level noise rescheduling) MDLM

Closest papers by meaning

• HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.89)
• Bridging Expressivity and Scalability with Adaptive Unitary SSMs (NeurIPS 2025 · similarity 0.87)
• Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models (ICLR 2025 · similarity 0.87)
• Elastic Spectral State Space Models for Train-Once Budgeted Inference (NeurIPS 2026 · similarity 0.86)
• Sequence Modeling with Spectral Mean Flows (NeurIPS 2025 · similarity 0.86)
• StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization (ICML 2024 · similarity 0.86)