Research map / Architectures and efficiency / CNN design

Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learning

ICML 2024 · Architectures and efficiency › CNN design · approach: Compact CNN architectures · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that L2-regularized CNNs learn a convolution bottleneck structure (keeping few Fourier frequencies inside the middle layers), proposes the CBN rank as the depth-scaled representation cost, and uses it to motivate down/up-sampling.

Problem

Understanding what structure CNNs learn under L2 regularization, in particular why middle representations are low-dimensional and supported on few frequencies, and whether this justifies down-sampling.

Approach

Defines a convolution bottleneck (CBN) rank as the sum over kept DFT frequencies of pooling-weighted costs, shows the large-depth representation cost R^(0) is upper-bounded by it (conjectured equal), and proves weights and activations of near-optimal CNNs have a bottleneck in Fourier frequencies, extending FC-network bottleneck results to CNNs.

Key results

• Trained MNIST classifier keeps only 6 constant-frequency dimensions in the bottleneck, enough to linearly separate 10 classes.
• Autoencoder on MNIST zeros keeps 3 constant, 4 degree-1 and 1 degree-2 frequencies.
• Theorem gives R(f;L) <= L*Rank_CBN(f) + c, and weight matrices of near-optimal networks are approximately supported on few DFT frequencies in at least (1-p)L layers.

Surprising findings

• A 'half-bottleneck' appears in classification: representations collapse to low-dim/low-frequency and stay there up to the output, which may relate to neural collapse.

Limitations

• Conjecture that R^(0) equals CBN rank is only partially proven.
• Analysis assumes full-size filters and cyclic boundaries, and no proof that gradient descent reaches near-optimal norm solutions.
• Bottleneck results rely on assumptions about parameter norm and Jacobian rank; activation bound requires large learning rate stability assumption.

Experimental setup

Datasets: MNIST

Benchmarks: MNIST

Metrics: singular value spectrum of pooled weights classification or reconstruction quality

Builds on: representation cost of FC networks neural tangent kernel bottleneck structure in FC networks

Closest papers by meaning

• Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.86)
• Structured IB: Improving Information Bottleneck with Structured Feature Learning (AAAI 2025 · similarity 0.85)
• Understanding and Leveraging the Learning Phases of Neural Networks (AAAI 2024 · similarity 0.85)
• Asymptotics of Learning with Deep Structured (Random) Features (ICML 2024 · similarity 0.85)
• Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.85)
• Generalization Bounds for Rank-sparse Neural Networks (NeurIPS 2025 · similarity 0.85)