Research map / Architectures and efficiency / RNNs and normalization

Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: LSTM design and memory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Recursive Quadrature Filters (RQFs), a band-pass special case of diagonal SSMs, and a parameter-free two-tap prospective input correction that improves deep stacks under full BPTT and mitigates depth attenuation under spatial-only backprop.

Problem

Deep continuous-time recurrent networks delay bottom-up signals and attenuate top-down errors through cascaded temporal integration, hurting learning on long sequences.

Approach

Replace each layer's instantaneous bottom-up input with a prospective (look-ahead) input chi_tau = 1 + tau d/dt, discretized as a parameter-free two-tap update within a diagonal SSM; RQFs are shown to be a constrained diagonal SSM with a phase-rotation prior.

Key results

• Non-residual width-32 six-layer RQF reaches 96.09±0.23% on raw-audio Speech Commands with 31.9k parameters.
• Width-64 six-layer RQF reaches 83.56±2.11% on Path-X (16,384 steps) under full BPTT, vs 81.63±0.80% for non-prospective.
• Under spatial-only backprop on MFCC, prospective advantage grows from 0.69 to 3.59 to 15.17 points with depth 2, 4, 6.

Surprising findings

• Under spatial-only backpropagation, the non-prospective six-layer raw-audio RQF is worse than the four-layer one's prospective counterpart by large margins, showing spatial attenuation with depth.
• Prospective ORGaNICs ties non-prospective at six layers under full BPTT on MFCC.

Limitations

• Not directly applicable to discrete-time gated RNNs such as LSTMs and GRUs.
• Spatial-only backprop still requires weight transport and is not biologically plausible training.
• Hyperparameters such as look-ahead scale not optimized; only fixed tau=5h tested.

Experimental setup

Datasets: Google Speech Commands v0.02 (10-class subset) Path-X (LRA)

Benchmarks: Speech Commands raw audio Speech Commands MFCC Path-X

Metrics: test accuracy (mean ± sample std over five seeds)

Baselines: native S5 alphaP-S5 non-prospective RQF ORGaNICs (instantaneous input)

Builds on: S5 S4D ORGaNICs Mamba-3 alphaP-S5

Closest papers by meaning

• Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers (NeurIPS 2021 · similarity 0.86)
• Twin Networks: Matching the Future for Sequence Generation (ICLR 2018 · similarity 0.85)
• Exploring the Promise and Limits of Real-Time Recurrent Learning (ICLR 2024 · similarity 0.85)
• Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity (ICML 2025 · similarity 0.84)
• FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks (NeurIPS 2025 · similarity 0.84)
• Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions (ICLR 2025 · similarity 0.84)