Research map / Architectures and efficiency / RNNs and normalization

Batch normalization is sufficient for universal function approximation in CNNs

ICLR 2024 · Architectures and efficiency › RNNs and normalization · approach: Batch normalization improvements

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves that training only normalization parameters (e.g., batch normalization scale and shift) of randomly initialized CNNs is sufficient for universal function approximation, given enough available random features.

Problem

Explaining why normalization layers contribute so significantly to deep learning success, specifically whether training only normalization parameters suffices for expressive power.

Approach

Theoretical result showing that, if the number of available (potentially random) features is sufficiently large, tuning only the batch normalization parameters of a network is already enough for universal function approximation in CNNs.

Key results

• Proves that training normalization parameters alone is sufficient for universal function approximation in CNNs when the number of available random features is large enough.

Closest papers by meaning

• Understanding Batch Normalization (NeurIPS 2018 · similarity 0.93)
• Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.91)
• Four Things Everyone Should Know to Improve Batch Normalization (ICLR 2020 · similarity 0.91)
• Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.89)
• Batch Normalization Is Blind to the First and Second Derivatives of the Loss (AAAI 2024 · similarity 0.88)
• Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.88)