Research map / Architectures and efficiency / RNNs and normalization
Batch Normalization Alleviates the Spectral Bias in Coordinate Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that adding batch normalization to coordinate networks shifts the NTK eigenvalue distribution upward, alleviating spectral bias and improving representation and inverse problems (image compression, CT, shape, MRI, NeRF).
Problem
Coordinate networks (MLPs mapping coordinates to signal values) suffer from spectral bias, learning high-frequency components slowly because of a pathological NTK eigenvalue distribution.
Approach
Applying batch normalization to coordinate networks reduces the max and variance of NTK eigenvalues while barely changing the mean, which shifts most eigenvalues upward and alleviates spectral bias; the analysis uses mean field theory and is combined with positional encoding.
Key results
- BN improves ReLU coordinate network PSNR by up to 4.39 dB on Kodak image fitting (10x40 config) and up to 4.75 dB (13x49).
- BN+PE achieves the highest PSNR among seven methods on 2D image, CT, and MRI tasks in Table 1 (e.g., 33.64 dB on CT 8x256).
- BN-enhanced DINER reaches 33.31 PSNR on the Blender NeRF benchmark, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add batch normalization to ReLU (ablation) | Coordinate network · 2D Image, [13×49] | PSNR: 24.33 -> 29.08 | Helped |
| Add batch normalization to positional-encoding ReLU (ablation) | PE coordinate network · 2D Image, [13×49] | PSNR: 28.23 -> 30.84 | Helped |
| Add batch normalization to ReLU (ablation) | Coordinate network · 2D CT, [8×256] | PSNR: 29.93 -> 31.24 | Helped |
| Add batch normalization to positional-encoding ReLU (ablation) | PE coordinate network · 2D CT, [8×256] | PSNR: 30.79 -> 33.64 | Helped |
| Add batch normalization to ReLU (ablation) | Coordinate network · 3D Shape, [2×256] | IoU: 0.904 -> 0.958 | Helped |
| Add batch normalization to positional-encoding ReLU (ablation) | PE coordinate network · 3D Shape, [2×256] | IoU: 0.968 -> 0.992 | Helped |
| Add batch normalization to ReLU (ablation) | Coordinate network · 3D MRI, [4×256] | PSNR: 26.11 -> 30.62 | Helped |
| Add batch normalization to positional-encoding ReLU (ablation) | PE coordinate network · 3D MRI, [4×256] | PSNR: 32.12 -> 34.56 | Helped |
| Add batch normalization (ablation) | NeRF · 5D NeRF | PSNR: 31.04 -> 31.37 | Helped |
| Add batch normalization (ablation) | DINER · 5D NeRF | PSNR: 33.11 -> 33.31 | Helped |
| Add batch normalization (ablation) | DINER · 5D NeRF | LPIPS: 0.034 -> 0.033 | Helped |
| Add batch normalization (ablation) | DINER · 5D NeRF | SSIM: 0.967 -> 0.968 | Helped |
Surprising findings
- BN alone with ReLU outperforms several frequency-based activation methods (SIREN, MFN, WIRE) in CT and MRI tasks, where SIREN fails to reconstruct effectively at deeper configurations.
Limitations
- Theoretical analysis assumes BN applied to the last layer; extension to deep BN networks is argued by stacking rather than fully proven.
- Analysis is statistical (mean, variance, max eigenvalue) and relies on mean field approximations at large width.
Experimental setup
Datasets: Kodak x-ray colorectal CT dataset Stanford 3D Scanning Repository ATLAS brain MRI Blender (NeRF synthetic)
Benchmarks: Kodak x-ray colorectal CT Stanford 3D Scanning Repository ATLAS brain MRI Blender
Metrics: PSNR SSIM LPIPS IoU
Baselines: ReLU MLP PE (positional encoding) SIREN MFN WIRE NeRF DINER Plenoxels DVGO
Builds on: neural tangent kernel batch normalization NeRF DINER positional encoding
Closest papers by meaning
- On the Frequency-bias of Coordinate-MLPs (NeurIPS 2022 · similarity 0.87)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.86)
- Riemannian approach to batch normalization (NeurIPS 2017 · similarity 0.86)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.85)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.84)
- Batch Normalization Is Blind to the First and Second Derivatives of the Loss (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.