Research map / Architectures and efficiency / Pruning and efficient tuning

AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaCubic, a second-order optimizer that adapts the cubic regularization weight via a dual/auxiliary cubic-constrained subproblem and Hutchinson diagonal Hessian estimation, evaluated against SGD, Adam, and AdaHessian on vision, NLU, LM, and camera-model identification tasks.

Problem

Cubic regularized Newton methods are theoretically appealing for escaping saddle points but the cubic weight is hard to select and their Hessian cost is prohibitive for deep learning.

Approach

Reformulates the cubic subproblem as a constrained problem with a cubic-norm constraint, derives strong duality so the Lagrange multiplier serves as an adaptive cubic weight M, and updates the trust-region-like parameter xi by acceptance ratios, using a Hutchinson diagonal Hessian approximation for scalability.

Key results

• On CIFAR-10 AdaCubic reaches 91.49% (ResNet20) and 92.65% (ResNet32), trailing AdaHessian (91.64%, 93.15%) slightly.
• On CIFAR-100 ResNet18 AdaCubic gets 71.81% without spatial averaging vs SGD 72.62%, and 72% with spatial averaging.
• On WikiText-2 with BERT, AdaCubic gets perplexity 5.759 vs SGD 13.380 and AdaHessian 16.151; on GLUE it is generally second-best behind SGD.

Surprising findings

• On GLUE AdaCubic often trails the SGD baseline, which was tuned per task, while AdaCubic uses one fixed hyperparameter set.
• Reported time to reach a target loss for AdaCubic (42.40 min) exceeds SGD (35.16 min) despite fewer epochs.

Limitations

• Relies on diagonal Hessian approximation, which may miss off-diagonal curvature, explaining gaps on CIFAR-100 and Transformer tasks.
• Memory footprint in practice is higher than the O(d) theoretical value due to PyTorch gradient retention; needs extra backward pass.
• Baselines like SGD are fine-tuned per task while AdaCubic uses universal hyperparameters; fair comparison limited to few seeds and settings.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 GLUE WikiText-2 PTB VISION

Benchmarks: CIFAR-10 CIFAR-100 GLUE WikiText-2 PTB VISION

Metrics: accuracy F1 Pearson correlation Spearman correlation perplexity training loss

Baselines: SGD Adam AdaHessian

Builds on: AdaHessian Adaptive Regularized Cubic (ARC) Cubic regularized Newton method Hutchinson's method Adam

Closest papers by meaning

• Efficient Hyper-parameter Optimization with Cubic Regularization (NeurIPS 2023 · similarity 0.88)
• Improving Deep Learning Optimization through Constrained Parameter Regularization (NeurIPS 2024 · similarity 0.87)
• Improving Generalization and Convergence by Enhancing Implicit Regularization (NeurIPS 2024 · similarity 0.86)
• Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.86)
• AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.85)
• Avoiding spurious sharpness minimization broadens applicability of SAM (ICML 2025 · similarity 0.85)