Research map / Optimization / Optimizers
CRONOS: Enhancing Deep Learning with Scalable GPU Accelerated Convex Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CRONOS, a GPU-accelerated ADMM with Nyström-preconditioned CG solver for convex reformulations of two-layer ReLU networks, and CRONOS-AM, which alternates CRONOS with DAdapted-Adam to train deeper networks, scaling to ImageNet and IMDb.
Problem
Convex reformulations of two-layer ReLU networks are theoretically appealing but the resulting constrained high-dimensional problems are too expensive to solve on realistic large datasets, and deep networks lack convex training with global guarantees.
Approach
Solve the convex reformulation with ADMM where the u-subproblem linear system is solved by Nyström-preconditioned conjugate gradient using only matrix-vector products (exploiting the structure of the activation-pattern matrices), implemented in JAX on GPU; extend to deep networks by alternating CRONOS on the last two layers with a non-convex optimizer on earlier layers.
Key results
- CRONOS-AM reaches 88.47% peak validation accuracy on binary ImageNet, comparable to tuned Shampoo (89.72%) and above DAdam (87.96%).
- CRONOS reaches 93.91% validation accuracy on IMDb after one epoch of AdamW fine-tuning of GPT-2, about 0.3% above tuned AdamW.
- On CIFAR-10 binary, CRONOS-AM attains 90.5% peak validation accuracy, versus 90.84% for best tuned Yogi, with no learning-rate search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CRONOS-AM instead of DAdam | 4-layer ReLU MLP · CIFAR-10 | peak validation accuracy: 90.15% -> 90.5% | Helped |
| Use CRONOS-AM instead of the best-tuned optimizer | 4-layer ReLU MLP · ImageNet | peak validation accuracy: 88.47% vs. 89.72% | Hurt |
| Use CRONOS instead of tuned AdamW | GPT2-NFT · IMDb | peak validation accuracy: 77.66% vs. [40.29,73.69]% | Helped |
| Use CRONOS instead of tuned AdamW | GPT2-FT · IMDb | peak validation accuracy: 93.91% vs. [48.30,93.69]% | Helped |
| Use CRONOS-AM instead of standard optimizers | 4-layer ReLU MLP · CIFAR-10 | runtime: 3.00 vs. 3.02; 3.00 vs. 2.28 | Mixed |
Surprising findings
- Shampoo outperforms CRONOS-AM on ImageNet by a fair margin, which the authors attribute to its approximate second-order nature.
- CRONOS-AM's runtime is higher than Adam and SGD on ImageNet (5.10 s vs 1.84 s for Adam).
Limitations
- Binary classification only for most vision experiments, and binary/downsampled ImageNet rather than full multi-class ImageNet.
- Convergence guarantee for CRONOS-AM with alternating minimization is not established.
- Shampoo outperforms CRONOS-AM on ImageNet; CRONOS-AM is slower than some first-order baselines on ImageNet.
Experimental setup
Datasets: CIFAR-10 ImageNet Food-5k Fashion-MNIST IMDb
Benchmarks: CIFAR-10 ImageNet IMDb
Metrics: peak validation accuracy runtime (s)
Baselines: SGD with momentum Adam AdamW Shampoo Yogi DAdapted-Adam r-FISTA
Builds on: NysADMM NysPCG DAdapted-Adam convex reformulation of two-layer ReLU networks
Closest papers by meaning
- CrAM: A Compression-Aware Minimizer (ICLR 2023 · similarity 0.86)
- The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions (ICLR 2022 · similarity 0.85)
- Scaling Convex Neural Networks with Burer-Monteiro Factorization (ICLR 2024 · similarity 0.85)
- DeepZero: Scaling Up Zeroth-Order Optimization for Deep Model Training (ICLR 2024 · similarity 0.85)
- Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions (ICML 2022 · similarity 0.85)
- Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.