Research map / Architectures and efficiency / RNNs and normalization
Neural Tangent Knowledge Distillation for Optical Convolutional Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NTKD, a task- and hardware-agnostic pipeline that distills digital teacher networks into hybrid optical-digital neural networks by matching neural tangent kernels, plus NTK-based performance estimation and NTK-guided backend fine-tuning to compensate fabrication errors.
Problem
Hybrid optical neural networks (optical frontend plus digital backend) suffer an accuracy gap versus large digital networks and further degradation from discrepancies between simulated and fabricated optics, and prior end-to-end methods are dataset- and hardware-specific.
Approach
Align the NTK matrices of an optical student and a pretrained digital teacher on minibatches (combined with end-to-end loss), use NTK regression to estimate achievable accuracy before training, and reuse NTK matching to fine-tune only the digital backend after fabrication.
Key results
- Simulated polychromatic CIFAR-10 classification improves from 56.4% (no transfer) and 72.5% (KD) to 75.6% with NTKD.
- On fabricated systems, NTKD error compensation raises monochromatic MNIST accuracy to 95.1% from 89.2% baseline and polychromatic segmentation mIoU to 81.2% from 49.7%.
- Hybrid ONN energy is 5.83 mJ per image versus 9.73 mJ for the compact digital network, a reported over 40% system-level reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Match the ONN and teacher NTKs instead of using prediction-only KD | Simulated monochromatic ONN · MNIST | accuracy: 95.9 ± 0.6% -> 97.3 ± 0.6% | Helped |
| Add NTK matching to training instead of using no transfer | Simulated monochromatic ONN · MNIST | accuracy: 91.4 ± 0.8% -> 97.3 ± 0.6% | Helped |
| Match the ONN and teacher NTKs instead of using prediction-only KD | Simulated polychromatic ONN · CIFAR-10 | accuracy: 72.5 ± 2.1% -> 75.6 ± 0.9% | Helped |
| Match the ONN and teacher NTKs instead of using prediction-only KD | Simulated Extreme Meta ONN · Carvana Image Masking | mIoU: 75.3 ± 0.2% -> 80.1 ± 0.2% | Helped |
| Match the ONN and teacher NTKs instead of using prediction-only KD | Simulated Polychromatic Meta ONN · Carvana Image Masking | mIoU: 86.7 ± 0.4% -> 91.5 ± 0.4% | Helped |
| Fine-tune the backend using NTK matching rather than end-to-end compensation | Fabricated monochromatic ONN · MNIST | accuracy: 93.2 ± 0.1% -> 95.1 ± 0.1% | Helped |
| Fine-tune the backend using NTK matching rather than end-to-end compensation | Fabricated polychromatic ONN · CIFAR-10 | accuracy: 70.4 ± 2.1% -> 74.9 ± 1.3% | Helped |
| Fine-tune the backend using NTK matching rather than end-to-end compensation | Fabricated Polychromatic Meta ONN · Carvana Image Masking | mIoU: 62.7 ± 0.9% -> 81.2 ± 0.6% | Helped |
| Increase random PSF kernel count from 8 to 1000 (ablation) | Random-kernel ONN · CIFAR-10 | accuracy: 36.73% -> 56.23% | Helped |
| Increase random PSF kernel count from 8 to 1000 (ablation) | Random-kernel ONN · Carvana Image Masking | mIoU: 49.3% -> 69.9% | Helped |
| Increase random PSF kernel count from 8 to infinitely many in the NTK regime (ablation) | Random-kernel ONN · CIFAR-10 | accuracy: 36.73% -> 67.33% | Helped |
| Increase random PSF kernel count from 8 to infinitely many in the NTK regime (ablation) | Random-kernel ONN · Carvana Image Masking | mIoU: 49.3% -> 72.1% | Helped |
Surprising findings
- Random, non-designed PSF kernels with infinite kernels (NTK estimate) reach 67.33% on polychromatic CIFAR-10, which the authors note still underperforms designed kernels with knowledge transfer.
- Reported ONN-based compressibility: 98 convolutional parameters achieve 95.2% MNIST accuracy, close to the NTK-predicted accuracy.
Limitations
- Current ONN architectures are shallow and linear, limiting accuracy relative to deep digital networks.
- Strong digital backends needed for robustness reduce the optical frontend's contribution and increase power use.
- Polychromatic fabrication suffers chromatic aberration and kernel mismatch, leaving a gap to simulation (e.g., 74.9% vs 75.6% and 81.2% vs 91.5% on segmentation).
Experimental setup
Datasets: MNIST CIFAR-10 Carvana Image Masking ImageNet-100
Benchmarks: MNIST CIFAR-10 Carvana Image Masking ImageNet-100
Metrics: classification accuracy mIoU standard deviation over 5 seeds MACs energy per image
Baselines: Simulation, No Transfer (end-to-end) Simulation, KD ExtremeMETA Compressed Meta ONN Polychromatic Meta ONN
Builds on: knowledge distillation neural tangent kernel regression Compressed Meta ONN Polychromatic Meta ONN ExtremeMETA
Closest papers by meaning
- Physics-Constrained Comprehensive Optical Neural Networks (NeurIPS 2024 · similarity 0.88)
- Layered-Parameter Perturbation for Zeroth-Order Optimization of Optical Neural Networks (AAAI 2025 · similarity 0.84)
- All-Optical Nonlinear Diffractive Deep Network for Ultrafast Image Denoising (CVPR 2025 · similarity 0.84)
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.84)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.83)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.