Research map / Bandits and online learning / Online convex optimization and games
A General Regret Bound of Preconditioned Gradient Method for DNN Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AdaBK preconditioning to SGDM | SGDM · CIFAR100, ResNet18 and ResNet50 | testing accuracy (%): ↑2.10; ↑3.48 | Helped |
| Add AdaBK preconditioning to AdamW | AdamW · CIFAR100, ResNet18 and ResNet50 | testing accuracy (%): ↑1.43; ↑2.05 | Helped |
| Add AdaBK preconditioning to SGDM | SGDM · CIFAR10, ResNet18 and ResNet50 | testing accuracy (%): ↑0.34; ↑1.11 | Helped |
| Add AdaBK preconditioning to AdamW | AdamW · CIFAR10, ResNet18 and ResNet50 | testing accuracy (%): ↑0.42; ↑0.68 | Helped |
| Add AdaBK preconditioning to SGDM | SGDM · ImageNet-1k val, ResNet18 and ResNet50 | top 1 accuracy (%): ↑1.10; ↑1.31 | Helped |
| Add AdaBK preconditioning to AdamW | AdamW · ImageNet-1k val, ResNet18 and ResNet50 | top 1 accuracy (%): ↑1.62; ↑1.10 | Helped |
| Add AdaBK preconditioning to AdamW | AdamW · ImageNet-1k val, Swin-T and Swin-B | top 1 accuracy (%): ↑0.61; ↑0.12 | Helped |
| Add AdaBK preconditioning to SGDM or AdamW | SGDM and AdamW · COCO val2017, Faster-RCNN R50 | AP: ↑2.2; ↑1.6 | Helped |
| Add AdaBK preconditioning to SGDM or AdamW | SGDM and AdamW · COCO val2017, Mask-RCNN R50 | APb: ↑2.2; ↑2.2 | Helped |
| Use the AdaBK technique | SGDM and AdamW training · CIFAR100 and COCO training curves | training loss | Helped |
Closest papers by meaning
- AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.88)
- Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks (ICLR 2018 · similarity 0.88)
- Explicitly Imposing Constraints in Deep Networks via Conditional Gradients Gives Improved Generalization and Faster Convergence (AAAI 2019 · similarity 0.88)
- Matrix-Free Preconditioning in Online Learning (ICML 2019 · similarity 0.88)
- AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (NeurIPS 2020 · similarity 0.87)
- On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.