Research map / Optimization / Optimizers
Deep Learning as a Mixed Convex-Combinatorial Optimization Problem
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use soft hinge per-layer losses with feasible target propagation instead of SSTE | 4-layer convnet with sign activations · CIFAR-10 | top-1 test accuracy: 80.6 -> 81.3 | Helped |
| Use soft hinge per-layer losses with feasible target propagation instead of SSTE | 4-layer convnet with 2-bit qReLU · CIFAR-10 | top-1 test accuracy: 85.6 -> 85.5 | No clear effect |
| Use soft hinge per-layer losses with feasible target propagation instead of SSTE | 8-layer convnet with sign activations · CIFAR-10 | top-1 test accuracy: 84.6 -> 84.9 | Helped |
| Use soft hinge per-layer losses with feasible target propagation instead of SSTE | 8-layer convnet with 2-bit qReLU · CIFAR-10 | top-1 test accuracy: 88.4 -> 89.8 | Helped |
| Use soft hinge per-layer losses with feasible target propagation instead of SSTE | AlexNet with sign activations · ImageNet | top-1 test accuracy: 46.7 -> 47.3 | Helped |
| Use soft hinge per-layer losses with feasible target propagation instead of SSTE | AlexNet with 2-bit qReLU · ImageNet | top-1 test accuracy: 59.4 -> 60.7 | Helped |
| Use soft hinge per-layer losses with feasible target propagation instead of SSTE | ResNet-18 with sign activations · ImageNet | top-1 test accuracy: 49.1 -> 47.8 | Hurt |
| Use soft hinge per-layer losses with feasible target propagation instead of SSTE | ResNet-18 with qReLU · ImageNet | top-1 test accuracy: 60.6 -> 64.3 | Helped |
| Use saturated ReLU instead of ReLU | AlexNet · ImageNet | top-1 test accuracy: 61.3 -> 61.9 | Helped |
| Use saturated ReLU instead of ReLU | ResNet-18 · ImageNet | top-1 test accuracy: 69.1 -> 66.9 | Hurt |
Closest papers by meaning
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.89)
- Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks (ICLR 2018 · similarity 0.87)
- Globally Optimal Training of Neural Networks with Threshold Activation Functions (ICLR 2023 · similarity 0.87)
- From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference (ICLR 2019 · similarity 0.87)
- A Convergence Theory for Deep Learning via Over-Parameterization (ICML 2019 · similarity 0.87)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.