Research map / Learning theory and algorithms / Kernels and regression theory
Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PEAK, a plug-in learnable polynomial input encoding plus a kernel alignment regularizer (KAR) that pushes an INR's NTK toward the optimal kernel, improving INR generalization on inverse problems.
Problem
INR generalization depends heavily on architecture, and there is no principled method to design architectures; the NTK of an INR is hard to match to the optimal kernel that minimizes pointwise expected squared error.
Approach
Shows that the NTK of an INR composed with a learnable input feature map can approximate any positive semidefinite dot-product kernel, and adds a Kernel Alignment Regularizer that enforces the INR output to be a softmax-weighted combination of its own training-point values (a Dirichlet-energy-like term), jointly optimized with a learnable polynomial encoder.
Key results
- On Baboon with random missing pixels, PEAK reaches 22.60 dB PSNR versus 18.64 (MLP), 20.10 (Fourier), 19.67 (Hash).
- In phase retrieval, PEAK improves PSNR by up to 8.69 dB in FPR and 11.68 dB in GPR.
- On NeRF with 25 views, PEAK outperforms Instant-NGP in average PSNR (e.g., 23.27 vs 21.74 on lego at 25 samples).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PEAK instead of Fourier encoding | INR image reconstruction · Baboon, random missing | PSNR (dB): 20.10 -> 22.60 | Helped |
| use PEAK instead of Hash encoding | INR image reconstruction · Cameraman, patch missing | PSNR (dB): 28.19 -> 35.20 | Helped |
| use PEAK instead of Hash encoding | INR in Net-ADM · House, FPR, s=1.9 | PSNR (dB): 31.75 -> 40.44 | Helped |
| use PEAK instead of Fourier encoding | INR in Net-GD · House, GPR, s=1.0 | PSNR (dB): 21.88 -> 33.19 | Helped |
| use PEAK instead of Instant-NGP | NeRF · chair, 50 views | PSNR (dB): 24.97 -> 28.34 | Helped |
| use PEAK instead of Instant-NGP | NeRF · chair, 100 views | PSNR (dB): 30.60 -> 30.32 | Hurt |
| vary polynomial order Nγ (ablation) | PEAK · Jetplane image reconstruction | PSNR (dB) | Mixed |
| tune λ to between 10−2 and 10−1 (ablation) | PEAK · Jetplane image reconstruction | PSNR (dB): around 23.5dB | Helped |
| set λ to 102 (ablation) | PEAK · Jetplane image reconstruction | PSNR (dB): around 14dB | Hurt |
| increase regularization network output dimension from r=10 to r=100 (ablation) | PEAK · Jetplane image reconstruction | PSNR (dB): from 19.5dB to 23.8dB | Helped |
| increase regularization network output dimension beyond r=100 (ablation) | PEAK · Jetplane image reconstruction | PSNR (dB): decreasing to around 22dB at r = 500 | Hurt |
| replace the activation with Softmax rather than other tested activations (ablation) | PEAK · Jetplane image reconstruction | PSNR (dB) | Helped |
Surprising findings
- Instant-NGP outperforms PEAK on the chair scene at 100 views (30.60 vs 30.32 dB).
- ReLU and LeakyReLU activations gave slightly lower performance than Softmax and Sigmoid in the activation ablation.
Limitations
- Equation (11) is only a necessary condition for the optimal INR; not all minimizers are optimal.
- Theoretical kernel construction assumes unit-norm inputs and a dot-product kernel; the NTK result is derived for ReLU and only conjectured for other activations.
- Computing the optimal kernel K* is intractable in practice, so alignment relies on a surrogate softmax similarity.
Experimental setup
Datasets: Baboon Boat Cameraman Jetplane Lake Livingroom House Boston NeRF synthetic dataset
Benchmarks: Image reconstruction with missing pixels (random, patch, textural) Fourier and Gaussian phase retrieval NeRF synthetic views
Metrics: PSNR
Baselines: MLP (ReLU) Fourier feature networks Hash encoding SIREN Instant-NGP Net-GD Net-ADM
Builds on: Neural Tangent Kernel Fourier feature networks Instant-NGP Net-GD Net-ADM
Closest papers by meaning
- Implicit Representations via Operator Learning (ICML 2024 · similarity 0.87)
- Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations (ICML 2025 · similarity 0.87)
- MIRE: Matched Implicit Neural Representations (CVPR 2025 · similarity 0.87)
- VI^3NR: Variance Informed Initialization for Implicit Neural Representations (CVPR 2025 · similarity 0.87)
- Grids Often Outperform Implicit Neural Representation at Compressing Dense Signals (NeurIPS 2025 · similarity 0.86)
- Towards Croppable Implicit Neural Representations (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.