Research map / Optimization / Optimizers
Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
Juhan Bae, Roger B Grosse
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add linearization of the hypernetwork (Centered STN to Delta-STN) (ablation) | Centered STN · CIFAR10 VGG16 | validation loss: 0.286 -> 0.272 | Improved |
| Add linearization (Centered STN to Delta-STN) (ablation) | Centered STN · CIFAR10 AlexNet | validation loss: 0.431 -> 0.425 | Improved |
| Add linearization (Centered STN to Delta-STN) (ablation) | Centered STN · CIFAR10 ResNet18 | validation loss: 0.222 -> 0.204 | Improved |
| Add linearization (Centered STN to Delta-STN) (ablation) | Centered STN · MNIST | validation loss: 0.041 -> 0.040 | Improved |
| Add linearization (Centered STN to Delta-STN) (ablation) | Centered STN · PTB | validation perplexity: 69.40 -> 68.63 | Improved |
| Replace STN with centered parameterization and modified training objective (ablation) | STN · CIFAR10 VGG16 | validation loss: 0.330 -> 0.286 | Improved |
| Replace STN with centered parameterization and modified objective (ablation) | STN · CIFAR10 AlexNet | validation loss: 0.474 -> 0.431 | Improved |
| Replace STN with centered parameterization and modified objective (ablation) | STN · CIFAR10 ResNet18 | validation loss: 0.266 -> 0.222 | Improved |
| Replace STN with centered parameterization and modified objective (ablation) | STN · PTB | validation perplexity: 70.67 -> 69.40 | Improved |
| Replace STN with centered parameterization and modified objective (ablation) | STN · FMNIST | test loss: 0.218 -> 0.212 | Improved |
| Replace STN with Delta-STN | STN · MNIST | test loss: 0.041 -> 0.038 | Improved |
| Replace Bayesian optimization with Delta-STN | Bayesian optimization · CIFAR10 VGG16 | validation loss: 0.421 -> 0.272 | Improved |
Closest papers by meaning
- Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis (ICLR 2024 · similarity 0.87)
- Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis (ICLR 2025 · similarity 0.86)
- A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness (ICML 2024 · similarity 0.86)
- BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimization (NeurIPS 2026 · similarity 0.86)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.86)
- Fast and Slow Gradient Approximation for Binary Neural Network Optimization (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
