Research map / Architectures and efficiency / RNNs and normalization
ReLUs Are Sufficient for Learning Implicit Neural Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that ReLU networks with constrained, B-spline-wavelet-shaped groups of seven ReLUs can learn state-of-the-art implicit neural representations, and uses the ReLU variation norm to quantify regularity and select the scaling hyperparameter c.
Problem
ReLU networks are poorly suited to implicit neural representations because their spectral bias and ill-conditioned feature Gram matrices slow gradient-based training on high-frequency, low-dimensional signals.
Approach
Constrain each group of seven ReLU neurons to share input weights and fixed relative biases so that together they realize a second-order B-spline wavelet activation; this yields a well-conditioned Gram matrix (condition number O(1)) while remaining exactly representable by a ReLU network. The variation norm of the resulting network then gives a principled way to choose the scaling parameter c.
Key results
- BW-ReLU performs on par with or slightly better than SIREN, WIRE and ReLU+P.E. on CT reconstruction, signal representation and 4x super resolution.
- In the CT inverse problem, the scaling c=2 gives the highest PSNR (32.1 dB) and the lowest variation norm across layers, versus 28.7-29.9 dB for other c values at equal training loss.
- Theory: the Gram matrix of ReLU neurons has condition number Omega(K^3), while the B-spline wavelet construction gives an O(1) condition number.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use constrained BW-ReLU neurons instead of standard ReLU neurons | INR models · Computed tomography reconstruction | PSNR | Helped |
| use constrained BW-ReLU neurons instead of standard ReLU neurons | INR models · Signal representation (Cameraman) | PSNR | Helped |
| use constrained BW-ReLU neurons instead of standard ReLU neurons | INR models · 4× super resolution (DIV2K) | PSNR | Helped |
| vary the activation scaling parameter c (ablation) | BW-ReLU network · CT reconstruction | PSNR | Mixed |
| vary the activation scaling parameter c (ablation) | BW-ReLU network · Super resolution | PSNR | Mixed |
| vary the activation scaling parameter c (ablation) | BW-ReLU network · Alternative CT reconstruction image | PSNR | Mixed |
Surprising findings
- A plain ReLU network with no positional encoding, once constrained to B-spline wavelet groups, matches state-of-the-art INR architectures that use sine or Gaussian-type activations.
- The scaling parameter c, usually viewed as a heuristic, has a clear effect on regularity: the c giving the best reconstruction is the one with the lowest variation norm, even though all networks reach the same training loss.
Limitations
- Experiments are limited to three INR tasks on single images; neural radiance fields and physics-informed networks are left to future work.
- The BW-ReLU network is restricted to a fixed architecture with three hidden layers and the scaling hyperparameter c must still be tuned per task.
- The condition-number result is proven for a specific dyadic neuron arrangement and a univariate setting; the multivariate case relies on the empirical results.
Experimental setup
Datasets: chest X-ray image cameraman image DIV2K Butterfly image
Benchmarks: CT reconstruction (chest X-ray, 100 projections) signal representation (cameraman 256x256) 4x super resolution (DIV2K)
Metrics: PSNR standard error across five trials variation norm
Baselines: SIREN WIRE ReLU + positional encoding ReLU
Builds on: SIREN WIRE positional encoding ridgelets
Closest papers by meaning
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.87)
- Implicit Representations via Operator Learning (ICML 2024 · similarity 0.87)
- MIRE: Matched Implicit Neural Representations (CVPR 2025 · similarity 0.87)
- A sampling theory perspective on activations for implicit neural representations (ICML 2024 · similarity 0.87)
- Activation-Descent Regularization for Input Optimization of ReLU Networks (ICML 2024 · similarity 0.87)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.