Research map / Architectures and efficiency / RNNs and normalization

Infinite-Dimensional Feature Interaction

NeurIPS 2024 · Architectures and efficiency › RNNs and normalization · approach: Mechanistic circuit interpretability · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InfiNet, a vision backbone whose InfiBlock uses an RBF kernel over multi-branch convolutional features to create an infinite-dimensional feature interaction space, reporting state-of-the-art ImageNet, COCO, and ADE20K results.

Problem

Element-wise multiplication in neural networks captures only finite-order feature interactions, and explicitly expanding interaction order is costly; the paper asks how to extend interactions to an infinite-dimensional space efficiently.

Approach

Replace the element-wise multiplication or addition interaction with an RBF kernel K(Z_a, Z_b)=exp(-||Z_a-Z_b||^2/2) applied to multi-branch convolutional features, whose RKHS is infinite-dimensional, so interactions span all monomial orders at the cost of one exponential.

Key results

• InfiNet-T reaches 83.4% top-1 on ImageNet-1K with 23M params and 3.2G FLOPs, versus ConvNeXt-T at 82.1% with 29M and 4.5G.
• InfiNet-XL (ImageNet-22K pretrained, fine-tuned at 384^2) reaches 88.2% top-1 on ImageNet-1K.
• InfiNet-XL achieves 56.3 box AP and 48.9 mask AP with Cascade Mask R-CNN, above ConvNeXt-L's 54.8 and 47.6.

Measured effects reported in the paper

Surprising findings

• Ablation shows the swap from additive to Hadamard to polynomial kernels yields gains of only 0.5-1.8 points before the RBF kernel reaches 83.4%, a notably larger step than the finite-order variants.

Limitations

• Only the RBF kernel and finite-degree polynomial kernels were tested; other kernels were not explored.
• Evaluated only on basic vision tasks in supervised training; no language or self-supervised validation.
• RBF bandwidth sigma fixed to 1 without tuning; no error bars due to training cost.

Experimental setup

Datasets: ImageNet-1K ImageNet-22K MS COCO ADE20K CIFAR10 Tiny-ImageNet

Benchmarks: ImageNet-1K MS COCO ADE20K

Metrics: top-1 accuracy box AP mask AP mIoU FLOPs params

Baselines: ConvNeXt SLaK Conv2Former UniFormer CoAtNet FocalNet Swin Transformer HorNet MogaNet

Builds on: ConvNeXt HorNet Swin Transformer Star Operation Mercer's theorem

Closest papers by meaning

• On the Joint Interaction of Models, Data, and Features (ICLR 2024 · similarity 0.83)
• Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.83)
• Monitoring Primitive Interactions During the Training of DNNs (AAAI 2025 · similarity 0.82)
• Learning Input Encodings for Kernel-Optimal Implicit Neural Representations (ICML 2025 · similarity 0.82)
• Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.81)
• Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs (NeurIPS 2025 · similarity 0.81)