Research map / Optimization / Optimizers
Fast Training of Large Kernel Models with Delayed Projections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EigenPro 4, a kernel-machine solver that delays the projection back to the model-center span for T batches, cutting per-iteration cost to amortized linear in model size and reporting large per-epoch speedups over EigenPro 3 and FALKON.
Problem
Training kernel machines with large model size (many centers) and large datasets is limited: FALKON needs quadratic memory in model size, and EigenPro 3 has quadratic per-iteration cost in model size due to a projection after every batch.
Approach
Delay EigenPro 3's projection step for T mini-batches, accumulating temporary centers and an accumulated gradient, then perform one projection; choosing T proportional to p/m makes amortized per-batch cost O(p).
Key results
- EigenPro 4 reports up to 686x speedup over EigenPro 3 per epoch at model size 1M on ImageNet-type features, with 21 minutes versus more than 10 days.
- At p=1M on Librispeech, EigenPro 4 reaches 89.5% accuracy in 70 minutes, a 411x speedup over EigenPro 3 (more than 20 days).
- FALKON runs out of memory beyond 256K centers on Webvision, CIFAR5M and Librispeech, while EigenPro 4 scales to 1M centers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use delayed projections instead of projecting every iteration | EigenPro4 vs EigenPro3 · Librispeech (p = 1M) | training speed: > 411x | Helped |
| Use delayed projections instead of projecting every iteration | EigenPro4 vs EigenPro3 · CIFAR5M* (p = 1M) | accuracy: 88.56% -> 88.7% | Mixed |
| Use delayed projections instead of projecting every iteration | EigenPro4 vs EigenPro3 · CIFAR5M (p = 1M) | accuracy | Helped |
| Use delayed projections instead of projecting every iteration | EigenPro4 vs EigenPro3 · Webvision (p = 1M) | accuracy | Helped |
Surprising findings
- EigenPro 4 converges in fewer epochs than EigenPro 3, reaching convergence on ImageNet in about 15 epochs versus about 100 for EigenPro 3 at 512K centers.
Limitations
- Projection is inexact and accuracy drops after each projection step, though the drop shrinks with training.
- Experiments are restricted to Laplace kernel and features extracted by pretrained networks rather than end-to-end training.
- Multi-GPU or distributed parallelization is left to future work.
Experimental setup
Datasets: CIFAR5M ImageNet Webvision Librispeech CIFAR10
Benchmarks: CIFAR5M ImageNet Webvision Librispeech
Metrics: test accuracy training time per epoch speedup over EigenPro 3
Baselines: EigenPro 3.0 FALKON
Builds on: EigenPro 2.0 EigenPro 3.0 FALKON
Closest papers by meaning
- Toward Large Kernel Models (ICML 2023 · similarity 0.91)
- Preconditioning Kernel Matrices (ICML 2016 · similarity 0.89)
- Learning Curves of Stochastic Gradient Descent in Kernel Regression (ICML 2025 · similarity 0.86)
- Generalization in Kernel Regression Under Realistic Assumptions (ICML 2024 · similarity 0.85)
- Improving Adaptivity via Over-Parameterization in Sequence Models (NeurIPS 2024 · similarity 0.85)
- Statistical Guarantees for High-Dimensional Stochastic Gradient Descent (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.