Research map / Architectures and efficiency / RNNs and normalization

A Renormalization Group Framework for Scale-Invariant Feature Learning in Deep Neural Networks (Student Abstract)

AAAI 2025 · Architectures and efficiency › RNNs and normalization · approach: Symmetry and equivariance in loss landscapes · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a theoretical renormalization-group framework for deep networks, with scale-aware activations, RG beta functions for weights and biases, and an RG-guided training procedure, without empirical validation.

Problem

Understanding and enforcing scale invariance in hierarchical feature learning of deep neural networks, which prior RG-deep learning connections treated only qualitatively or for specific architectures.

Approach

Views layer-wise transformations as RG coarse-graining steps, defines scale-aware activation functions with scale covariance, derives beta functions for weights and biases, and adds an RG flow term to gradient descent so training converges to scale-invariant fixed points.

Key results

• Theorem 2 derives beta functions β(W_i)=(α−1)W_i and β(b_i)=−αb_i for scale-aware layers.
• Theorem 3 and Theorem 4 claim fixed points correspond to scale-invariant representations and that RG-guided training converges to them; both are supported only by proof sketches.
• Theorem 6 claims a mapping between NN RG flow and the Ising model near criticality, given only as a sketch.

Limitations

• No empirical validation; the paper states that validation through simulations is future work.
• Theorems rely on proof sketches rather than full proofs.
• Scale-aware activation and the RG step are not tested on any dataset or architecture.

Experimental setup

Builds on: renormalization group theory data processing inequality

Closest papers by meaning

• MS$^3$D: A RG Flow-Based Regularization for GAN Training with Limited Data (ICML 2024 · similarity 0.86)
• Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.86)
• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.85)
• Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.85)
• Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.84)
• The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks (NeurIPS 2024 · similarity 0.84)