Research map / Architectures and efficiency / CNN design

Is Kernel Prediction More Powerful than Gating in Convolutional Neural Networks?

ICML 2024 · Architectures and efficiency › CNN design · approach: Large and dynamic kernel design

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically that gating layers combined with fixed-weight layers can express HyperNetwork weight prediction (and that low-rank HyperNetworks equal factorization machines), then uses a gated double layer to replace kernel prediction in a burst denoising KPN with lower memory.

Problem

Whether weight prediction (HyperNetworks, kernel prediction) is strictly more powerful than gating in the hierarchy of multiplicative interactions, given that gating layers are virtually always used with adjacent fixed-weight layers.

Approach

Gated double layers (fixed linear layer, element-wise multiplication with a predicted activation, fixed linear layer) can exactly reproduce any HyperNetwork layer when the latent dimension is at least the product of input and output sizes, so the multiplicative-interaction hierarchy is flat once fixed layers are counted; a low-rank HyperNetwork is equivalent to a factorization machine.

Key results

• A gated double layer with latent size at least n_i*n_j exactly reproduces any HyperNetwork layer (Theorem 3.1), so the multiplicative-interaction hierarchy is flat when fixed layers are included.
• On burst denoising, the gated-128 KPN variant reaches 32.8 test PSNR versus 32.6 for the original KPN, with its widest layer about 1.5x smaller.
• Gated-64 drops to 31.9 PSNR, showing that too-aggressive shrinking of the hidden layer hurts quality.

Measured effects reported in the paper

Surprising findings

• Gating with fixed layers matches weight prediction in expressivity, so the claimed hierarchy of multiplicative interactions is flat in practice.
• The gated-128 variant slightly outperforms the original KPN while using less memory.

Limitations

• No proof that SGD will reliably learn a good gated double layer when a good HyperNetwork is learnable.
• Generalization to multi-headed attention based architectures is not straightforward.
• Theoretical equivalence relies on construction with large latent dimension; low-rank variants may lose quality.

Experimental setup

Datasets: CIFAR-10 McMaster

Benchmarks: CIFAR-10 emulation relative error synthetic burst denoising test set (McMaster)

Metrics: relative L2 error MSE test PSNR max layer size

Baselines: Original KPN MLP matched in FLOPs Single Frame

Builds on: Kernel Prediction Networks HyperNetworks Factorization Machines hierarchy of multiplicative interactions

Closest papers by meaning

• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed (ICML 2021 · similarity 0.86)
• Deep Convolutional Networks as shallow Gaussian Processes (ICLR 2019 · similarity 0.86)
• End-to-End Kernel Learning with Supervised Convolutional Kernel Networks (NeurIPS 2016 · similarity 0.85)
• More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity (ICLR 2023 · similarity 0.85)
• On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.84)
• Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.84)