Research map / Architectures and efficiency / CNN design
Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MRConv, a reparameterized multi-resolution global convolution with learnable kernel decay and low-rank (Fourier, dilated, sparse) sub-kernels, evaluated on LRA, sCIFAR, Speech Commands and ImageNet.
Problem
Training long global convolution kernels as sequence models is hard and prone to overfitting; existing parameterizations (SSMs, implicit kernels, SGConv) are either complex or lose performance relative to SSMs.
Approach
Construct long kernels as a learnable, BatchNorm-normalized sum of multi-resolution low-rank sub-kernels of increasing length but fixed parameter count, trained in parallel via structural reparameterization and merged into one convolution at inference; uses Fourier, dilated, or sparse sub-kernel parameterizations.
Key results
- MRConv-L achieves 88.20 average accuracy on Long Range Arena, the best among sub-quadratic models in the paper's table.
- MRConv with dilated kernels reaches 94.26% on sCIFAR, improving on MultiresNet's 93.15%.
- MRConvNeXt-S with Fourier+Sparse kernels reaches 83.4% top-1 on ImageNet, matching SGConvNeXt-S and above ConvNeXt-S's 83.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ConvNeXt-T 2D convolutions with 1D MRConv layers | ConvNeXt-T · ImageNet | Top-1 accuracy: 82.1 -> 82.2 | Helped |
| Replace ConvNeXt-S 2D convolutions with 1D MRConv layers | ConvNeXt-S · ImageNet | Top-1 accuracy: 83.1 -> 83.4 | Helped |
| Replace ConvNeXt-B 2D convolutions with 1D MRConv layers | ConvNeXt-B · ImageNet | Top-1 accuracy: 83.8 -> 83.9 | Helped |
| Use MRConv with dilated kernels | sCIFAR models · sCIFAR | accuracy: 93.15 -> 94.26 | Helped |
| Use Fourier kernels in MRConv | MRConv · Speech Commands 8 kHz | accuracy: 94.53 -> 95.05 | Helped |
| Add multiresolution to a dense kernel (ablation) | Dense-kernel MRConv · LRA ListOps | accuracy: 49.25 -> 53.15 | Helped |
| Add multiresolution to a dense kernel (ablation) | Dense-kernel MRConv · LRA Image | accuracy: 82.90 -> 86.65 | Helped |
| Add Fourier kernel parameterization (ablation) | Multiresolution MRConv · LRA ListOps | accuracy: 53.15 -> 57.05 | Helped |
| Add Fourier kernel parameterization (ablation) | Multiresolution MRConv · LRA Image | accuracy: 86.65 -> 86.19 | Hurt |
| Add BatchNorm (ablation) | Multiresolution MRConv with Fourier kernels · LRA ListOps | accuracy: 57.05 -> 62.40 | Helped |
| Add BatchNorm (ablation) | Multiresolution MRConv with Fourier kernels · LRA Image | accuracy: 86.19 -> 88.55 | Helped |
| Concatenate sub-kernels instead of summing them (ablation) | MRConv-Base, Fourier · LRA Path-X | accuracy: 50.00 -> 96.19 | Helped |
Surprising findings
- Dilated kernels reach SOTA on sCIFAR but the Fourier-based variant is better on Speech Commands and LRA average, and Path-X accuracy was random for all dilated and sparse variants.
- Learning the linear combination of multi-resolution sub-kernels caused overfitting on Path-X; concatenating sub-kernels with fixed decay worked better there.
Limitations
- Parallel multi-branch training increases memory and training time because BatchNorm is nonlinear during training.
- Model is input-independent, so it underperforms linear-time transformers on information-dense discrete text tasks.
- Does not support fast autoregressive inference by construction, unlike SSMs.
Experimental setup
Datasets: Long Range Arena sCIFAR Speech Commands ImageNet
Benchmarks: Long Range Arena sCIFAR Speech Commands ImageNet
Metrics: test accuracy throughput FLOPs
Baselines: S4 S4D S5 Liquid-S4 SGConv MEGA MEGA-Chunk SPADE-Chunk BST MultiresNet CCNN ConvNeXt
Builds on: SGConv S4 structural reparameterization (RepVGG) ConvNeXt RepLKNet
Closest papers by meaning
- What Makes Convolutional Models Great on Long Sequence Modeling? (ICLR 2023 · similarity 0.87)
- FlexConv: Continuous Kernel Convolutions With Differentiable Kernel Sizes (ICLR 2022 · similarity 0.86)
- Simple Hardware-Efficient Long Convolutions for Sequence Modeling (ICML 2023 · similarity 0.86)
- FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores (ICLR 2024 · similarity 0.85)
- Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling (NeurIPS 2024 · similarity 0.85)
- KernelWarehouse: Rethinking the Design of Dynamic Convolution (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.