Research map / Optimization / Implicit bias and generalization
Nonparametric Teaching of Implicit Neural Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Casts implicit neural representation training as nonparametric teaching and proposes Implicit Neural Teaching (INT), a greedy selection of high-error pixels/samples that cuts INR training time by 30%+.
Problem
Training overparameterized MLP-based implicit neural representations (INRs) is costly for high-resolution signals, since every iteration uses all coordinates.
Approach
Shows that the dynamic NTK of an overparameterized MLP converges to the canonical kernel of functional gradient descent, so MLP training aligns with nonparametric teaching; then applies a greedy teacher that selects the training examples with the largest signal-fit disparity, with an incremental selection ratio and interval schedule.
Key results
- INT reduces encoding time by 31.63% on audio, 38.88% on 2D images (Kodak, PSNR 36.09 to 36.97 dB), and 35.54% on 3D shapes (IoU drop under 1%).
- On the megapixel Pluto image (8192x8192), INT cuts training time by 29.26% (16.78K s to 11.87K s) with PSNR 31.82 to 33.01 dB.
- Among INT schedules on an 8-image Kodak subset, incremental ratio plus incremental interval achieves the best time (211.04 s vs 345.22 s baseline) with PSNR 37.04 dB vs 35.95 dB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use incremental INT instead of no INT | INR signal fitting · Librispeech audio | Time (s): 23.05 -> 15.76 (-31.63%) | Helped |
| Use incremental INT instead of no INT | INR signal fitting · Kodak images | PSNR (dB): 36.09$\pm$2.51 -> 36.97$\pm$3.59 | Helped |
| Use incremental INT instead of no INT | INR signal fitting · Pluto megapixel image | PSNR (dB): 31.82 -> 33.01 | Helped |
| Use incremental INT instead of no INT | INR signal fitting · Stanford 3D Scanning Repository | IoU (%): 97.07$\pm$0.84 -> 96.68$\pm$0.83 | Hurt |
| Incrementally increase sampling ratio and interval (ablation) | INT · Kodak (8 representative images) | Time (s): 345.22 -> 211.04 | Helped |
| Use reverse-cosine ratio and decremental interval schedules (ablation) | INT · Kodak (8 representative images) | PSNR (dB): 35.18$\pm$1.44 -> 33.56$\pm$2.53 | Hurt |
| Use reverse-cosine ratio and decremental interval schedules (ablation) | INT · Kodak (8 representative images) | SSIM: 0.920$\pm$0.02 -> 0.894$\pm$0.03 | Hurt |
| Sample densely every iteration instead of using incremental intervals (ablation) | INT with incremental ratio · Kodak (8 representative images) | Time (s): 211.04 -> 468.01 | Hurt |
| Use cosine ratio scheduling instead of incremental ratio with incremental intervals (ablation) | INT · Kodak (8 representative images) | PSNR (dB): 36.61$\pm$2.55 -> 37.04$\pm$2.51 | Hurt |
Surprising findings
- Decremental selection ratio and interval severely degrades reconstruction quality (PSNR 33.56 dB), attributed to the spectral bias of INRs toward learning low-to-high frequencies.
- INT trained on the Cameraman image produces visible background artifacts from over-emphasis on boundary pixels, which a progressively increasing selection rate mitigates.
Limitations
- Selection requires inferring all data through the network at each selection step, which adds overhead that the scheduling only partly offsets.
- Greedy selection over-emphasizes high-error boundary pixels, causing artifacts on flat backgrounds with a fixed low selection ratio.
- The NTK-to-canonical-kernel convergence theorem relies on convexity assumptions and an infinite-width limit that is not strictly satisfied by the dynamic-width MLPs used.
Experimental setup
Datasets: Librispeech Kodak Pluto Stanford 3D Scanning Repository Cameraman
Benchmarks: Librispeech Kodak Pluto Stanford 3D Scanning Repository Cameraman
Metrics: training time (s) PSNR (dB) SSIM IoU (%)
Baselines: SIREN without INT random sampling at 20% rate Cosine INT schedule R-Cosine INT schedule Incremental INT schedule
Builds on: nonparametric teaching iterative machine teaching SIREN neural tangent kernel parametric teaching
Closest papers by meaning
- Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context (ICML 2024 · similarity 0.86)
- Implicit Neural Representations and the Algebra of Complex Wavelets (ICLR 2024 · similarity 0.85)
- VI^3NR: Variance Informed Initialization for Implicit Neural Representations (CVPR 2025 · similarity 0.84)
- Implicit Representations via Operator Learning (ICML 2024 · similarity 0.84)
- Few-shot Implicit Function Generation via Equivariance (CVPR 2025 · similarity 0.84)
- MIRE: Matched Implicit Neural Representations (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.