Research map / Architectures and efficiency / RNNs and normalization
Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Inductive Gradient Adjustment (IGA), which uses an empirical NTK computed on a sparse subset of samples to build a gradient transformation matrix that is generalized to the full data, to counter spectral bias in MLP-based implicit neural representations.
Problem
Spectral bias of MLPs makes them slow to fit high-frequency content in implicit neural representations, and the full NTK is intractable and quadratic in data size, so existing training-dynamics fixes (Fourier reparameterization, batch norm) lack reliable guidance.
Approach
Justify the empirical NTK as a tractable surrogate for the NTK in linear dynamics, then compute the eNTK on one sampled point per group of coordinates (largest residual) and inductively apply the resulting eigenvalue-balancing gradient transform to all points in the group, controlling spectral bias via the number of balanced eigenvalues.
Key results
- On 2D image approximation, PE+IGA reaches 32.46 average PSNR vs 28.64 for PE vanilla and 31.65 for PE+FR, with SSIM 0.8822 vs 0.7832.
- On NeRF (Blender), NeRF+IGA achieves 31.47 PSNR vs 31.23 for vanilla NeRF, 31.35 for NeRF+FR.
- IGA averages 1.56x training time and 1.34x memory of the baseline across experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IGA to PE | PE · Kodak 24 first 8 images | PSNR: 28.64 -> 32.46 | Helped |
| Add IGA to PE | PE · Kodak 24 first 8 images | SSIM: 0.7832 -> 0.8822 | Helped |
| Add IGA to PE | PE · Kodak 24 first 8 images | MS-SSIM: 0.9466 -> 0.9752 | Helped |
| Add IGA to PE | PE · Kodak 24 first 8 images | LPIPS: 0.2223 -> 0.0938 | Helped |
| Add IGA to NeRF | NeRF · Blender down-scaled dataset | SSIM: 0.953 -> 0.955 | Hurt |
| Add IGA to NeRF | NeRF · Blender down-scaled dataset | PSNR: 31.23 -> 31.47 | Helped |
| Add IGA to NeRF | NeRF · Blender down-scaled dataset | LPIPS: 0.029\textsuperscript{Alex} -> 0.027\textsuperscript{Alex} | Helped |
| Increase the number of balanced eigenvalues (ablation) | IGA · 1D function approximation, ReLU and SIREN | Relative error | Helped |
| Increase group size p (ablation) | IGA · 1D function approximation, ReLU | Relative error | No clear effect |
| Use IGA with p=1 instead of p=4 or p=8 (ablation) | IGA · 1D function approximation | Relative error | Mixed |
Surprising findings
- BN is reported as incompatible with SIREN, so SIREN+BN was omitted from comparisons.
Limitations
- Full eNTK is infeasible for large data (over 80 GiB memory), so IGA relies on sampling and approximation with theoretical guarantees only asymptotic in network width.
- Theory is limited to a two-layer network with fixed last layer; the general case is supported only empirically.
Experimental setup
Datasets: Kodak 24 3D shape dataset (five objects) Blender dataset synthetic 1D and S^1 functions
Benchmarks: Kodak 24 Blender
Metrics: PSNR SSIM MS-SSIM LPIPS IOU Chamfer distance
Baselines: ReLU MLP PE SIREN Fourier reparameterization (FR) batch normalization (BN) NeRF
Builds on: Neural tangent kernel linear dynamics Fourier reparameterized training batch normalization for INRs SIREN NeRF
Closest papers by meaning
- Improved Implicit Neural Representation with Fourier Reparameterized Training (CVPR 2024 · similarity 0.88)
- VI^3NR: Variance Informed Initialization for Implicit Neural Representations (CVPR 2025 · similarity 0.87)
- Learning Input Encodings for Kernel-Optimal Implicit Neural Representations (ICML 2025 · similarity 0.87)
- FINER: Flexible Spectral-bias Tuning in Implicit NEural Representation by Variable-periodic Activation Functions (CVPR 2024 · similarity 0.87)
- Implicit Representations via Operator Learning (ICML 2024 · similarity 0.86)
- Few-shot Implicit Function Generation via Equivariance (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.