Research map / Learning theory and algorithms / Kernels and regression theory
Learning via Surrogate PAC-Bayes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SuPAC, a surrogate PAC-Bayes learning framework that replaces the empirical risk in Catoni's bound by its L2 projection onto a low-dimensional functional space, enabling closed-form posterior updates with far fewer risk queries, and applies it to an ADM1 biochemical calibration task and meta-learning.
Problem
Optimising PAC-Bayes generalisation bounds is computationally expensive when each empirical risk query is costly (e.g., models requiring stiff ODE/PDE solves), since gradient estimation requires many new risk evaluations.
Approach
Replace the empirical risk in a PAC-Bayes bound by its orthogonal projection onto a finite-dimensional functional space (score approximation of dimension d+1 built from the score gradient), so that for Catoni's bound on exponential families the surrogate minimiser has a closed form; the projection is estimated from past risk evaluations via Voronoi-cell weighting.
Key results
- SuPAC-CE reduced the PAC-Bayes objective to 0.121 ± 0.004 (posterior risk 0.102 ± 0.003), similar to the prior work's 0.122 (risk 0.101) but reported as about twenty times faster.
- The average SuPAC-CE performance after 1800 risk queries beat the best GD performance after the full 9600 queries.
- In the meta-learning experiments, the objective dropped from 0.61 to 0.24 for lambda=0.1 and from 0.14 to 0.050 for lambda=0.01 after 150 meta gradient steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace gradient descent with SuPAC-CE | PAC-Bayes bound minimisation on the anaerobic digestion task · Anaerobic digestion (LF training set) | PAC-Bayes bound versus risk-query budget: average performance after 1800 queries better than the best performance obtained after 9600 queries for GD | Helped |
| use SuPAC-CE instead of the prior-work calibration | PAC-Bayes calibration on the anaerobic digestion task · Anaerobic digestion (LF training set) | PAC-Bayes bound: $0.122$ -> $0.121\pm 0.004$ | No clear effect |
| use SuPAC-CE instead of the prior-work calibration | PAC-Bayes calibration on the anaerobic digestion task · Anaerobic digestion (LF training set) | average empirical risk: $0.101$ -> $0.102 \pm 0.003$ | No clear effect |
| add meta-learning over the prior for 150 gradient steps | Meta-learning objective · Synthetic meta-learning test tasks, λ=0.1 | average generalisation bound: 0.61 -> 0.24 | Helped |
| add meta-learning over the prior for 150 gradient steps | Meta-learning objective · Synthetic meta-learning test tasks, λ=0.01 | average generalisation bound: 0.14 -> 0.050 | Helped |
| increase κmax to 10 (ablation) | SuPAC-CE · Anaerobic digestion (LF training set) | final PAC-Bayes bound: $0.147 \pm 0.022$ | Hurt |
| increase κmax to 10 (ablation) | SuPAC-CE · Anaerobic digestion (LF training set) | between-run standard deviation of final bound: $0.0032$ -> $0.061$ | Hurt |
| use κmax=0.01 and dampen=0.9 (ablation) | SuPAC-CE · Anaerobic digestion (LF training set), after 2000 risk queries | PAC-Bayes bound | No clear effect |
| use κmax=0.5, 1, or 2 and dampen=0.25, 0.5, or 0.75 (ablation) | SuPAC-CE · Anaerobic digestion (LF training set) | optimisation performance | No clear effect |
| add Nesterov momentum and compare with SuPAC-CE (ablation) | Gradient descent for PAC-Bayes bound minimisation · Anaerobic digestion (LF training set) | optimisation efficiency | Hurt |
| increase gradient-descent step size to twice the original with Nesterov momentum (ablation) | Nesterov accelerated gradient descent · Anaerobic digestion (LF training set) | optimisation stability: 3 out of 8 runs; 2 out of 8 runs | Hurt |
| use higher Nesterov momentum (ablation) | Nesterov accelerated gradient descent · Anaerobic digestion (LF training set) | runs reducing objective below 0.2: less than 3 runs out of 8 | Hurt |
Surprising findings
- 4 out of 20 GD runs ended worse than a single SuPAC-CE optimisation step.
- Nesterov accelerated GD, at the tested hyperparameters, was unable to match SuPAC-CE's speed or stability.
Limitations
- Practicable only when the parameter/predictor dimension is small (a few hundred at most); needs at least d+1 risk evaluations.
- Voronoi-cell weighting is unlikely to work in high-dimensional predictor spaces, ruling out deep learning settings.
- Choosing the surrogate validity range (KL step size, dampening) is a manual hyperparameter choice.
Experimental setup
Datasets: ADM1 anaerobic digestion calibration dataset 'LF' (training part) synthetic risk functions for meta-learning
Benchmarks: ADM1 anaerobic digestion calibration task (synthetic dataset from prior work) synthetic meta-learning tasks
Metrics: PAC-Bayes Catoni objective average posterior empirical risk number of risk queries to reach objective
Baselines: gradient descent Nesterov accelerated gradient descent Catoni bound posterior from prior work (ADM1 calibration)
Builds on: Catoni PAC-Bayes bound Maurer-Langford-Seeger bound gradient descent
Closest papers by meaning
- Learning via Wasserstein-Based High Probability Generalisation Bounds (NeurIPS 2023 · similarity 0.91)
- PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees (ICML 2021 · similarity 0.90)
- More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms (ICML 2024 · similarity 0.89)
- Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Loss (NeurIPS 2024 · similarity 0.88)
- Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks (ICML 2025 · similarity 0.87)
- Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.