Research map / Architectures and efficiency / RNNs and normalization

The Multiscale Single-Index Model: A Toy Model for Hierarchical Feature Learning

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies the Multiscale Single-Index Model (MSIM) as a toy for deep hierarchical feature learning, showing via Edgeworth expansions a finite-rank staircase structure in Wiener chaoses, depth-separation lower bounds against shallow nets, and online SGD recovery with about d^{K-1} samples.

Problem

Understanding how deep architectures learn hierarchical multiscale features with scale separation: when shallow networks fail, and whether plain online SGD on all layers at one timescale can recover the planted directions efficiently.

Approach

Uses a high-order Edgeworth expansion of the first Wiener chaos to reveal a finite-rank staircase hierarchy (not just an unstructured O(d^{-1/2}) perturbation of a spiked tensor), which allows overlap growth for arbitrary depth K and yields SGD recovery with O~(d^{K-1}) samples.

Key results

• Edgeworth expansion gives first-chaos remainder O(d^{-K/2}) with staircase levels of Frobenius norm d^{-q/2}.
• Online spherical SGD reaches min overlap >= epsilon within O~(d^{K-1}) samples and 1-o(1) overlap after an extra O(d^{K/2} log d) phase.
• Spectral tensor unfolding recovers directions with n >> d^{ceil(K/2)} log d samples.

Surprising findings

• For K>2, energy fluctuations of order d^{-1/2} dominate the correlation gradient, so SGD on the mean-squared error objective does not share the correlation objective's efficiency.

Limitations

• Analysis restricted to non-degenerate activations (information exponent 1) and incoherent (delocalized) planted directions.
• Recovery results hold conditional on a favorable initialization basin of constant-in-d probability.
• Whether the balanced-unfolding threshold is a computational barrier is left open.

Experimental setup

Metrics: overlap with planted directions sample complexity Frobenius approximation error

Baselines: Tensor PCA spectral methods shallow neural networks

Builds on: Tensor PCA multi-index models Ben Arous et al. information exponent

Closest papers by meaning

• Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.85)
• Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.85)
• The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.85)
• Learning single index models via harmonic decomposition (NeurIPS 2025 · similarity 0.85)
• Learning Orthogonal Multi-Index Models: A Fine-Grained Information Exponent Analysis (NeurIPS 2025 · similarity 0.85)
• Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.85)