Research map / Optimization / Optimizers
A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Turns bi-level optimization into a stochastic problem over an energy-based inner distribution, samples it with SGLD, and computes a memory-efficient forward-mode hypergradient recursion for meta learning and hyperparameter optimization.
Problem
Differentiable bi-level optimization (BLO) for meta learning and hyperparameter optimization struggles with inner-loop uncertainty (non-converged minibatch SGD, many inner minima from overparametrization), and existing hypergradient methods (IFT, RMD, FMD) are unstable or memory-infeasible at scale.
Approach
Recasts BLO as stochastic optimization where the inner loss defines a Gibbs distribution p(theta|lambda) with temperature tau, samples it via SGLD, and estimates the hypergradient with a forward recursion on g_m that uses a first-order approximation so that no Hessians or Jacobians are stored.
Key results
- On the L1-regularized ViT-B-16 HPO task, HPO-SGLD reaches about 23.75% to 24.8% test error on Oxford-Pets versus 25.49% for RMD and IFT variants that diverge or vary widely.
- On loss function learning, HPO-SGLD with tau=0.01 achieves 16.76% test error on CIFAR-10/AlexNet, compared to 23.62% for vanilla cross-entropy and 17.34% to 18.96% for IFT variants.
- On 5-way 1-shot MiniImageNet, HPO-SGLD achieves about 50.3% accuracy, above MAML (48.70%) and Reptile (49.97%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use stochastic optimization instead of deterministic BLO | tabular synthetic polynomial BLO · Synthetic polynomial hyperparameter search | optimal lambda found: three out of 20 different runs -> always found the optimal solution | Helped |
| Use HPO-SGLD rather than IFT-Neumann | noisy synthetic 1D problem · Noisy synthetic 1D | lambda* error: 0.0153 -> 0.0008 | Helped |
| Increase the SGLD temperature from tau=10^-10 to tau=10^-2 (ablation) | HPO-SGLD · Noisy synthetic 1D | lambda* error: 0.0019 -> 0.0008 | Helped |
| Use HPO-SGLD instead of RMD | L1-regularizer HPO with ViT-B-16 · Pets | test error (%): 25.49 ± 0.64 -> 23.75 ± 0.21 | Helped |
| Use HPO-SGLD instead of Taylor GLO | loss function learning with Alexnet · CIFAR-10 | test error (%): 20.99 ± 0.26 -> 16.76 ± 0.05 | Helped |
| Use HPO-SGLD instead of MAML | few-shot meta-learning · MiniImageNet 5-way 1-shot | meta test accuracy (%): 48.70 ± 1.84 -> 50.38 ± 0.87 | Helped |
| Use HPO-SGLD instead of Reptile | meta-learning implicit neural representations · ShapeNet Chairs | test PSNR: 18.85 -> 18.77 | Hurt |
| Use HPO-SGLD instead of Reptile | meta-learning implicit neural representations · ShapeNet Cars | test PSNR: 22.80 -> 23.16 | Helped |
| Use HPO-SGLD instead of Augerino | invariance learning · Rot-180 MNIST | test accuracy (%): 97.83 ± 0.03 -> 97.77 ± 0.03 | Hurt |
| Use HPO-SGLD instead of LILA | invariance learning · Rot-90 MNIST | test accuracy (%): 97.81 ± 0.11 -> 97.86 ± 0.04 | Helped |
| Increase MC sample count from M=1 to M=5 (ablation) | HPO-SGLD · Pets | test error (%): 25.17 ± 0.44 -> 24.04 ± 0.38 | Helped |
| Vary the HPO-SGLD temperature (ablation) | few-shot meta-learning · MiniImageNet 5-way 1-shot | meta test accuracy (%): 50.38 ± 0.87; 50.36 ± 0.83; 50.30 ± 0.87; 50.29 ± 0.84; 50.28 ± 0.83 | No clear effect |
Surprising findings
- On a grid-search toy problem, the deterministic BLO solution found the optimal lambda in only 3 of 20 runs, whereas stochastic optimization always did.
Limitations
- The first-order approximation and exactness results rely on infinitesimal step size assumptions and theta^(0) independent of lambda.
- FMD and RMD are infeasible or limited at the larger scales, so comparisons are restricted to smaller settings.
- Temperature and noise scale (tau, kappa) must be tuned, and the paper does not establish a general rule for choosing them.
Experimental setup
Datasets: Oxford-Pets DTD Flowers CIFAR-10 CIFAR-100 MiniImageNet ShapeNet MNIST (transformed)
Benchmarks: Oxford-Pets DTD Flowers CIFAR-10 CIFAR-100 MiniImageNet ShapeNet rotated/scaled/translated MNIST
Metrics: test error (%) test accuracy (%) PSNR absolute error of learned hyperparameters
Baselines: IFT-Neumann IFT-CG RMD FMD MAML FO-MAML Reptile iMAML Augerino LILA Taylor GLO AmIGO
Builds on: forward-mode differentiation reverse-mode differentiation implicit function theorem hypergradient MAML stochastic gradient Langevin dynamics
Closest papers by meaning
- Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization (NeurIPS 2024 · similarity 0.89)
- Learning to Explore for Stochastic Gradient MCMC (ICML 2024 · similarity 0.88)
- Gradient-based Hyperparameter Optimization Over Long Horizons (NeurIPS 2021 · similarity 0.88)
- MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning (AAAI 2024 · similarity 0.86)
- Memory-Reduced Meta-Learning with Guaranteed Convergence (AAAI 2025 · similarity 0.86)
- Adaptive Algorithms with Sharp Convergence Rates for Stochastic Hierarchical Optimization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.