Research map / Architectures and efficiency / Attention and transformers

Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Linear attention approximations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HyenaND, a subquadratic global operator using input-conditioned multi-dimensional long convolutions (SIREN kernels with FiLM conditioning from register tokens) plus a fused CUDA library nSubQ, evaluated on genomics, ImageNet, PDE modeling, and 3D medical segmentation.

Problem

Subquadratic operators like Mamba are inherently 1D and require rasterizing multi-dimensional data into ad-hoc scan orders, while attention is quadratic and forces patchification; standard long convolutions lack input dependence.

Approach

An N-dimensional long convolution whose kernel is generated by a joint N-D SIREN with learnable Gaussian window and FiLM conditioning on softmax-weighted register tokens, so the kernel is input-dependent per sample while keeping a single FFT path; fused CUDA kernels make it fast in wall-clock time.

Key results

• Pure HyenaND reaches 81.5% top-1 on ImageNet at patch 16x16, matching ViT-5-Small (81.8%).
• On PanTS 3D segmentation, the HHAA hybrid achieves mean Dice 0.7559 vs 0.7496 for the SwinUNETR attention baseline, with peak memory ~58.7 GB vs 65.79 GB.
• On genomics, the 2-MHA-layer hybrid H2 reaches mean validation perplexity 2.77 vs 2.92 for full transformer.

Surprising findings

• Attention performs near random on native-resolution spatial recall (e.g., 0.253 vs 0.284 random on 2D color_cond), attributed to low information density per pixel token.
• Mamba (bidir) degrades to near mean-predictor baseline on 3D simple_copy (0.451 vs 0.482).

Limitations

• Fused kernel speedups drop for sequence lengths around 16K-64K due to multi-kernel Cooley-Tukey path; not yet fully fused.
• Length-generalization bound is only meaningful for small deviations from training length; PDE results lag on some datasets with larger patches attributed to fixed frequency parameter.
• Code is omitted for double-blind review, so reproducibility is currently unverified.

Experimental setup

Datasets: ImageNet-1K OpenGenome2 The Well PanTS EMNIST (spatial recall tasks)

Benchmarks: ImageNet-1K OpenGenome2 validation perplexity The Well (VRMSE) PanTS Mean Dice spatial-recall MSE

Metrics: top-1 accuracy perplexity VRMSE Mean Dice MSE GFLOPs throughput peak memory

Baselines: ViT-5-Small attention Mamba-ViT Mamba (bidir) ConvNeXt U-Net (CNextU-net) SwinUNETR full transformer Evo2-style striped hybrid

Builds on: Hyena CKConv SIREN FiLM Evo2

Closest papers by meaning

• Geometric Hyena Networks for Large-scale Equivariant Learning (ICML 2025 · similarity 0.85)
• Implicit Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes (NeurIPS 2024 · similarity 0.85)
• Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.85)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.84)
• Explicitly Modeling Subcortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness (NeurIPS 2025 · similarity 0.84)
• Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.84)