Research map / Learning theory and algorithms / Kernels and regression theory
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add outer-loop SGLD noise to Entropy-SGD to obtain a differentially private data-dependent prior | Entropy-SGD · Two-class MNIST random labels, CONV | overfitting | Helped |
| Use τ = √m for the Entropy-SGLD prior | Entropy-SGLD · Two-class MNIST true labels, CONV | test error: approximately 2% | Mixed |
| Use τ = √m for the Entropy-SGLD prior | Entropy-SGLD · Two-class MNIST random labels, CONV | overfitting | Helped |
| Set β to 0.004 and use γ=3 while keeping βτ=2000 (ablation) | Entropy-SGLD · CIFAR10 true labels | test error: ≈ 0.19 | Helped |
| Use γ=0.003 with β=1 at fixed privacy (ablation) | Entropy-SGLD · CIFAR10 true labels | test error | Hurt |
| Increase τ above 10^5 (ablation) | Entropy-SGLD · CIFAR10 random labels | generalization gap | Hurt |
| Use γ=3 rather than γ=0.03 (ablation) | Entropy-SGLD · CIFAR10 random labels | overfitting: τ > 2*10^{2} compared to τ > 5*10^{4} | Hurt |
| Increase thermal noise (ablation) | SGLD · MNIST true and random labels | generalization error | Helped |
| Increase thermal noise (ablation) | Entropy-SGD · MNIST true and random labels | generalization error | Helped |
| Use thermal noise 0.01 instead of the very low-noise settings (ablation) | Entropy-SGD · MNIST random labels | overfitting: 0.01 | Helped |
| Use τ below 0.01, producing large prior variance and excessive smoothing (ablation) | Entropy-SGLD · CIFAR10 | test error: τ<0.01 | Hurt |
Closest papers by meaning
- Data-dependent PAC-Bayes priors via differential privacy (NeurIPS 2018 · similarity 0.90)
- A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descent (NeurIPS 2017 · similarity 0.89)
- Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processes (NeurIPS 2019 · similarity 0.88)
- Rethinking Information-theoretic Generalization: Loss Entropy Induced PAC Bounds (ICLR 2024 · similarity 0.88)
- Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Loss (NeurIPS 2024 · similarity 0.86)
- Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.