Research map / Optimization / Implicit bias and generalization
Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cone decomposition (CD-A approximation) after gated ReLU solve instead of exact SOCP (ablation) | C-ReLU via gated ReLU · energy UCI | test accuracy: same 86.3 as CD-SOCP | No clear effect |
| Replace exact cone decomposition SOCP with approximate CD-A (ablation) | cone decomposition step · energy UCI | time to solution: 134.6 s to 1.56 s | Helped |
| Accelerated R-FISTA with line-search and restarts for C-GReLU vs direct AL on C-ReLU | ReLU training · energy UCI | time (s): 0.12 s vs 5.05 s for AL | Helped |
| Convex gated ReLU training instead of SGD/Adam on non-convex problem | image classification (CIFAR-10) · CIFAR-10 gated ReLU | test accuracy: 56.4 vs 54.3 (SGD), 50.1 (Adam) | Helped |
| Convex gated ReLU training instead of non-convex optimizers | image classification (MNIST) · MNIST gated ReLU | test accuracy: 97.6 vs 98.0 (Adam), 97.2 (SGD), 97.5 (Adagrad) | Mixed |
| Convex C-GReLU solver vs SGD on NC-GReLU across UCI datasets | UCI classification (20 datasets) · UCI gated ReLU | median test accuracy: e.g. ecoli 82.1 vs SGD 4.5 (failure); magic 86.9 vs 82.1 | Mixed |
| Adding an acceleration-based R-FISTA (line-search, restarts, step-size init) to proximal gradient on C-GReLU | group-lasso C-GReLU · Appendix D.4 convergence study | convergence speed | Helped |
| Windowing heuristic for penalty delta in AL method (ablation) | AL method for C-ReLU · UCI optimization profile (73 datasets) | fraction of problems solved: fixed delta=1 or 1e4 fails on ~40% of problems | Helped |
| Sub-sampling activation patterns to a small set (10 to 1000) for C-ReLU/C-GReLU (ablation) | primary-tumor and other UCI datasets · UCI sensitivity | test accuracy distribution | No clear effect |
| Convex C-ReLU/C-GReLU vs random forests and SVMs on UCI | 18 UCI binary datasets · UCI classification | test accuracy: best on 9 datasets vs 4 for most competitive baseline | Helped |
Closest papers by meaning
- Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs (ICML 2021 · similarity 0.94)
- Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time (ICML 2024 · similarity 0.93)
- Convex Formulations for Training Two-Layer ReLU Neural Networks (ICLR 2025 · similarity 0.92)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.89)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.89)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.