Research map / Architectures and efficiency / Pruning and efficient tuning

Scalable Neural Network Kernels

ICLR 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces scalable neural network kernels (SNNKs) that replace feedforward layers by disentangling inputs and parameters via universal random features, and evaluates them as adapters, pooler replacements, and in toy/UCI settings.

Problem

Feedforward layers in neural networks have many trainable parameters and cost O(ld) per input; the paper seeks a compressed, computationally cheaper replacement that still approximates the layer.

Approach

Reinterpret a feedforward layer as a vector of dot-product kernel values between input embeddings and learnable parameter embeddings (SNNK), built from universal random features that linearize any activation with a Fourier transform (including sin/cos), so only m-dimensional parameter embeddings need training; also a ReLU-SNNK tied to arc-cosine kernels, and a bundling process that merges consecut

Key results

• ReLU-SNNK-Adapter outperforms the Houlsby adapter baseline on 5 of 8 GLUE dev tasks (e.g., RTE 69.68 vs 63.83, MRPC 91.26 vs 84.8) while using about 1/3 of trainable parameters.
• Cosine-SNNK-pooler beats the BERT pooler baseline on 5 of 8 GLUE tasks, reducing stored pooler+classifier storage from 18.92 Megabit to 0.02 Megabit.
• Bundled pooler+classifier training with 1024 random features uses about 1/30 of baseline trainable parameters, staying within about 2.7 points on MNLI.

Measured effects reported in the paper

Surprising findings

• Brute-force linearization of tanh (Tanh-SNNK-pooler) performs much worse than the baseline (e.g., RTE 52.08 vs 57.5), so the proposed URF/ReLU constructions are needed.
• The Cosine-SNNK-Adapter underperforms the Houlsby adapter on CoLA (52.12 vs 56.51) and MNLI (79.28 vs 83.53).

Limitations

• Replacing adapters or pooler with SNNKs is not uniformly better: several GLUE tasks (e.g., MNLI, QNLI, QQP) remain behind baselines unless more random features are used.
• Hyperparameters for SNNK adapters were borrowed from prior work and not tuned, as the authors acknowledge.
• Theoretical error propagation analysis for bundled networks is an informal bound with composed delta functions.

Experimental setup

Datasets: GLUE CIFAR-10 CIFAR-100 ImageNet-1k MNIST CoverType HIGGS HEPMASS

Benchmarks: GLUE dev set CIFAR-10 CIFAR-100 ImageNet-1k MNIST UCI datasets

Metrics: accuracy MCC F1 Spearman correlation trainable parameters

Baselines: BERT pooler baseline Houlsby adapter AdapterFusion-style adaptable adapters (AA) SIREN MLP baseline on MNIST 3-layer MLP on UCI

Builds on: random Fourier features FAVOR+ positive random features arc-cosine kernels adapters (Houlsby)

Closest papers by meaning

• Finite Versus Infinite Neural Networks: an Empirical Study (NeurIPS 2020 · similarity 0.87)
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel (NeurIPS 2020 · similarity 0.86)
• Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs (CVPR 2022 · similarity 0.86)
• Fixing the NTK: From Neural Network Linearizations to Exact Convex Programs (NeurIPS 2023 · similarity 0.86)
• Gradient Descent in Neural Networks as Sequential Learning in Reproducing Kernel Banach Space (ICML 2023 · similarity 0.86)
• EUGens: Efficient, Unified and General Dense Layers (NeurIPS 2025 · similarity 0.86)