Research map / Optimization / Nonconvex and smooth optimization
Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes risk-averse (CVaR) training for neural processes via a two-level finite-sum minimax reformulation solved with a variance-reduced, double-loop stochastic mirror prox algorithm.
Problem
Neural processes trained with empirical risk minimization treat tasks uniformly, so the worst fast-adaptation tasks can have catastrophic error in risk-sensitive settings.
Approach
Replace ERM with CVaR (expected tail risk) minimization over task losses, recast as a DRO minimax problem over task weights, and solve it with a variance-reduced stochastic mirror prox that samples one point per task and aggregates prox points with adaptive learning-rate weights.
Key results
- On CelebA, RANP reaches 2.9718 log-likelihood vs 2.9214 for ANP and RTNP 4.4226 vs TNP 4.4042 (seen classes), with consistent gains across the 15 NP variants.
- On Bayesian optimization with RBF kernel, RANP achieves regret 0.1025 vs ANP 0.1245; RTNP 0.0915 vs TNP 0.1125.
- On contextual bandits (gamma=0.999), RTNP attains cumulative regret 7.2355 vs TNP 9.5694, and RConvNP improves over ConvNP across gamma settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CVaR risk-averse optimization to ANP | ANP · CelebA | log-likelihood: 2.9214±0.004 -> 2.9718±0.006 | Helped |
| Apply CVaR risk-averse optimization to ANP | ANP · EMNIST seen (0-9) | log-likelihood: 0.9815±0.006 -> 0.9881±0.004 | Helped |
| Apply CVaR risk-averse optimization to ANP | ANP · EMNIST unseen (10-46) | log-likelihood: 0.8843±0.003 -> 0.8967±0.005 | Helped |
| Apply CVaR risk-averse optimization to TNP | TNP · Bayesian optimization, RBF | simple regret: 0.1125±0.003 -> 0.0915±0.004 | Helped |
| Apply CVaR risk-averse optimization to TNP | TNP · Bayesian optimization, Matérn 5/2 | simple regret: 0.1451±0.001 -> 0.1289±0.003 | Helped |
| Apply CVaR risk-averse optimization to TNP | TNP · Bayesian optimization, Periodic | simple regret: 0.1715±0.003 -> 0.1463±0.004 | Helped |
| Apply CVaR risk-averse optimization to neural processes | Neural processes · Contextual bandit (γ = 0.7) | cumulative regret: 1.1895±0.942 -> 1.1536±0.103 | Helped |
| Apply CVaR risk-averse optimization to neural processes | Neural processes · Contextual bandit (γ = 0.9) | cumulative regret: 1.4362±0.378 -> 1.5962±0.539 | Mixed |
| Apply CVaR risk-averse optimization to neural processes | Neural processes · Contextual bandit (γ = 0.95) | cumulative regret: 2.1526±0.574 -> 1.9854±0.341 | Helped |
| Apply CVaR risk-averse optimization to TNP | TNP · Contextual bandit (γ = 0.99) | cumulative regret: 2.3612±1.522 -> 2.0813±0.896 | Helped |
| Apply CVaR risk-averse optimization to TNP | TNP · Contextual bandit (γ = 0.995) | cumulative regret: 3.5267±1.267 -> 3.1231±0.957 | Helped |
| Apply CVaR risk-averse optimization to TNP | TNP · Contextual bandit (γ = 0.999) | cumulative regret: 8.9824±0.678 -> 7.2355±1.429 | Helped |
Surprising findings
- Reported that models using attention perform worse on contextual bandits than those without attention, contrary to other tasks.
- Bootstrapping BNP and BANP sometimes degrade bandit performance strongly (e.g., BNP regret ~58-62 across gamma).
Limitations
- Convergence guarantee only for convex settings; no general guarantee for deep non-convex NPs, as the authors acknowledge.
- Full-gradient snapshots in the outer loop are costly; mini-batch substitute is suggested but not analyzed.
- Experiments are on relatively small-scale synthetic and image benchmarks.
Experimental setup
Datasets: CelebA EMNIST synthetic GP functions (RBF, Matern 5/2, Periodic) contextual bandit synthetic data
Benchmarks: CelebA EMNIST Bayesian optimization GP kernels contextual bandits
Metrics: log-likelihood of target points simple regret cumulative regret task risk distribution (MSE)
Baselines: CNP NP ANP ConvNP TNP BCNP BNP BANP BConvNP BTNP SCNP SNP
Builds on: neural processes stable neural processes bootstrapping neural processes variance-reduced stochastic gradient (Johnson & Zhang, 2013) stochastic mirror prox (Alacaoglu & Malitsky, 2022)
Closest papers by meaning
- Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning (NeurIPS 2024 · similarity 0.86)
- Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments (ICML 2025 · similarity 0.85)
- Bayesian Optimization via Continual Variational Last Layer Training (ICLR 2025 · similarity 0.83)
- Gaussian Process Upper Confidence Bound Achieves Nearly-Optimal Regret in Noise-Free Gaussian Process Bandits (NeurIPS 2025 · similarity 0.83)
- Improved Regret Analysis in Gaussian Process Bandits: Optimality for Noiseless Reward, RKHS norm, and Non-Stationary Variance (ICML 2025 · similarity 0.83)
- Learning Scenario Reduction for Two-Stage Robust Optimization with Discrete Uncertainty (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.