Research map / Architectures and efficiency / Attention and transformers
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Orchid, a sub-quadratic sequence-mixing architecture whose core is a data-dependent global convolution with an input-conditioned kernel (shift-invariant conditioning networks), evaluated on associative recall, BERT-style language modeling (GLUE), and ViT image classification.
Problem
Attention has quadratic cost in sequence length, while static long convolutions lack data-dependent expressivity needed for in-context recall and long-context tasks.
Approach
Replace the static long-convolution kernel with one generated from the input by a conditioning network built from shift-invariant operations (magnitude of frequency-domain features, or frequency-domain cross-correlation of two short-convolved projections), so the data-dependent convolution stays shift-equivariant and runs in O(L log L) via FFT, combined with gating (element-wise multiplications) i
Key results
- Orchid-BERT-base (77M) reaches 80.6 average GLUE vs 79.6 for BERT-base (110M), a +1.0 gain with 30% fewer parameters.
- On ImageNet-1K, Orchid (48M) achieves 80.2 top-1 vs 78.5 for ViT-b (87M).
- On CIFAR-10 with 4x4 patches, Orchid-s (735K) reaches 88.5% top-1 vs 78.6% for ViT (1.2M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace attention layers with Orchid layers in BERT-base | BERT-base · GLUE | average GLUE score: 79.6 -> 80.6 | Helped |
| Replace attention layers with Orchid layers in BERT-large | BERT-large · GLUE | average GLUE score: 82.1 -> 82.7 | Helped |
| Replace ViT attention with Orchid | ViT-b · ImageNet-1K | top-1 accuracy: 78.5 -> 80.2 | Helped |
| Replace ViT attention with Orchid | ViT · CIFAR-10 (4 × 4 patches) | top-1 accuracy: 78.6 -> 88.5 | Helped |
| Use Type II cross-correlation conditioning instead of Type I (ablation) | Orchid-s · CIFAR-10 (2 × 2 patches) | top-1 accuracy: 92.2 -> 92.3 | Helped |
| Use 2 × 2 patches instead of 4 × 4 patches | Orchid · CIFAR-10 | top-1 accuracy | Helped |
| Use spatial-then-spectral Conv1D conditioning instead of spatial-only or spectral-only conditioning (ablation) | Orchid conditioning network · Associative Recall | test accuracy | Helped |
| Remove the nonlinearity from Type II cross-correlation conditioning (ablation) | Type II Orchid conditioning network · Associative Recall | test accuracy | Helped |
| Use Type I phase-suppression conditioning instead of Type II cross-correlation conditioning (ablation) | Orchid conditioning network · Associative Recall | convergence speed | Mixed |
| Use DCT instead of DFT (ablation) | Orchid conditioning network · Associative Recall | learning speed | Helped |
| Use orthonormal Fourier transforms instead of non-orthonormal transforms (ablation) | Orchid conditioning network · Associative Recall | learning speed | Helped |
Surprising findings
- Removing the nonlinearity in the cross-correlation conditioning network gave the best performance in the ablation, and DCT with orthonormal normalization converged faster than DFT.
- Orchid-BERT-large underperforms M2-BERT-large on CoLA (56.97 vs 59.2), which the authors attribute to lack of hyperparameter search.
Limitations
- The current formulation is not directly compatible with causal autoregressive (GPT-style) models.
- Hyperparameters for GLUE were reused from prior work without thorough search, and CoLA results trail M2-BERT.
- Runtime uses standard PyTorch FFT rather than fused or FlashFFT kernels, so speed gains are not fully realized.
Experimental setup
Datasets: C4 GLUE ImageNet-1K CIFAR-10 Speech Commands SC10 Associative recall (synthetic)
Benchmarks: GLUE ImageNet-1K CIFAR-10 Speech Commands SC10 Associative recall
Metrics: accuracy GLUE average score top-1 accuracy top-5 accuracy forward and backward runtime
Baselines: H3 CKConv Hyena BERT-base BERT-large M2-BERT-base M2-BERT-large ViT-b Hyena-ViT-b M2-ViT-b S4 FlashAttention
Builds on: Hyena CKConv Monarch Mixer BERT Vision Transformer
Closest papers by meaning
- Convolutional Sequence to Sequence Learning (ICML 2017 · similarity 0.88)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.85)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.85)
- Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling (NeurIPS 2024 · similarity 0.85)
- Revisiting Convolution Architecture in the Realm of DNA Foundation Models (ICLR 2025 · similarity 0.84)
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.