Research map / Learning theory and algorithms / Kernels and regression theory
Preconditioning Neural Tangent Kernel for Adaptive Optimization
Xiyuan Yang, Wenxuan Bao, Katherine Tieu, Jingrui He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Adam-style preconditioner estimation to NTK (OAK vs NTK) | RoBERTa NTK · MNLI/MRPC/QQP avg | accuracy: 47.38 -> 52.91 | Improved |
| add Adam-style preconditioner estimation to NTK (OAK vs NTK) | Qwen2.5-0.5B NTK · MNLI/MRPC/QQP avg | accuracy: 46.30 -> 48.28 | Improved |
| replace SignGD heuristic with estimated preconditioner | RoBERTa SignGD-NTK · MNLI/MRPC/QQP avg | accuracy: 52.20 -> 52.91 | Improved |
| replace SignGD heuristic with estimated preconditioner | Qwen2.5-0.5B SignGD-NTK · SST-2/CoLA avg | accuracy: 52.39 -> 54.27 | Improved |
| replace NTK with OAK | Qwen2.5-0.5B · SST-2/CoLA avg | accuracy: 54.13 -> 54.27 | No clear effect |
| add variance (AdaGrad) component to momentum-only OAK, i.e. full Adam (ablation) | OAK (SGDM) · MNLI n=64 | accuracy: 35.85 -> 37.66 | Improved |
| add momentum component to variance-only OAK, i.e. full Adam (ablation) | OAK (AdaGrad) · MNLI n=64 | accuracy: 37.34 -> 37.66 | Improved |
| use AdamW instead of Adam preconditioner | OAK · MNLI n=64 | accuracy: 37.66 -> 37.89 | No clear effect |
| use OAK-selected 50% subset instead of full data | RoBERTa finetuning · MNLI n=64 | accuracy: 35.5 -> 35.3 | No clear effect |
| use OAK instead of NTK for data selection | RoBERTa finetuning · MRPC n=64 subset | accuracy: 71.3 -> 72.8 | Improved |
| use OAK in low-data regime instead of finetuning | RoBERTa · SST-2 n=8 | accuracy: 52.28 -> 55.60 | Improved |
| increase finetuning samples n (proxy for training steps) | OAK vs finetuning gap · SST-2 Qwen2.5-0.5B | accuracy: 55.14 -> 57.69 | Hurt |
Closest papers by meaning
- Adaptive kernel predictors from feature-learning infinite limits of neural networks (ICML 2025 · similarity 0.86)
- On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.86)
- Fast Training of Large Kernel Models with Delayed Projections (NeurIPS 2025 · similarity 0.86)
- The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.86)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.86)
- Globally Convergent Variational Inference (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
