Research map / Optimization / Implicit bias and generalization

Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Neural tangent kernel analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Neural LoFi, a layerwise, backpropagation-free spectral surrogate for small-initialization gradient descent in which each layer keeps top eigendirections of a label-weighted second-moment operator and lifts them through a random feature map, analyzed theoretically and illustrated on CIFAR-10 and a synthetic hierarchical model.

Problem

Understanding which features deep networks discover during training, in what order, and at what sample complexity, and why depth helps learning hierarchical functions beyond the lazy/NTK regime.

Approach

Under layer-wise vanishing initialization, early gradient descent on each layer reduces to a linear dynamics driven by a label-weighted moment operator C = E[y z z^T]; Neural LoFi keeps its leading eigendirections (ordered by absolute eigenvalue) and lifts them via a fixed random feature map, iterating across layers to build task-adaptive kernels.

Key results

• On binary CIFAR-10 (animals vs vehicles), Neural LoFi improves over ridge regression and three-layer random-feature baselines, and in low-data/early-training regimes matches or exceeds layerwise and full-backprop GD test error.
• In the solvable hierarchical model, the Neural LoFi estimator shows a drop in test error near the predicted sample scale n ∝ d^2.5, with rising overlap to the planted representation.
• Eigenvector overlaps for a three-layer FCN on CIFAR-10 rise near the predicted emergence thresholds from the residual effective dimension criterion.

Surprising findings

• Applying the label-weighted operator directly to CIFAR-10 pixels recovers color, contrast and edge-like convolutional filters without any backpropagation.
• Best test error is not always achieved by retaining the largest number of features; the optimal number of retained features depends on sample size.

Limitations

• Derivation holds only in a stylized vanishing-initialization, small-time limit; the paper states it is a surrogate, not a replacement for backpropagation.
• Only second-order (degree-two) label-weighted operators are analyzed in depth; higher-order variants lead to difficult tensor spectral problems.
• Experiments are small-scale mechanistic illustrations on binary CIFAR-10 rather than competitive training results.

Experimental setup

Datasets: CIFAR-10

Benchmarks: CIFAR-10 binary animals vs vehicles

Metrics: test error MSE eigenvector overlap normalized subspace overlap gain

Baselines: ridge regression three-layer random features layerwise gradient descent full backpropagation gradient descent Recursive Feature Machines

Builds on: neural tangent kernel single-index and multi-index models rainbow networks average gradient outer product / Recursive Feature Machines compositional information exponent

Closest papers by meaning

• Deep Edge Filter: Return of the Human-Crafted Layer in Deep Learning (NeurIPS 2025 · similarity 0.87)
• Multiscale Low-Frequency Memory Network for Improved Feature Extraction in Convolutional Neural Networks (AAAI 2024 · similarity 0.84)
• The Master Key Filters Hypothesis: Deep Filters Are General (AAAI 2025 · similarity 0.84)
• Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.84)
• Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks (ICLR 2025 · similarity 0.84)
• PICNN: A Pathway towards Interpretable Convolutional Neural Networks (AAAI 2024 · similarity 0.84)