Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Rényi Neural Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rényi Neural Processes (RNP), which replace the KL term in NP variational and ML objectives with Rényi divergence to mitigate prior misspecification caused by coupling prior and posterior parameters.
Problem
Neural processes couple the parameters of the conditional prior and the approximate posterior, which amounts to prior misspecification and yields biased posterior variance and oversmoothed predictions.
Approach
Replace the KL divergence in NP objectives with the Rényi divergence (alpha in (0,1)), which reweights Monte Carlo samples by likelihood so a misspecified prior has less influence on posterior updates; the VI and ML objectives are recovered as alpha tends to 1 and 0 respectively.
Key results
- RNP improved context and target log-likelihood over VI/ML objectives for NP, ANP, BA-NP, TNP-D and VNP in most settings, e.g., NP RBF target LL 0.26 to 0.33 and ANP Periodic target -0.91 to -0.57.
- Under noisy contexts, RNP outperformed VI on all six datasets for NP and ANP (e.g., NP RBF -0.53 to -0.45).
- On EMNIST misspecified test (classes 11-46), TNP-D target LL improved from -0.20 to 0.70 with RNP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the standard NP objective with the Rényi objective | NP, ANP, BA-NP, TNP-D, and VNP · RBF, Matern 5/2, Periodic, MNIST, SVHN, and CelebA context and target sets | log-likelihood | Helped |
| Replace the standard objective with RNP | TNP-D · Hare-Lynx domain-shift context and target sets | log-likelihood: -0.59±0.47 -> -0.17±0.31; -4.44±0.41 -> -3.63±0.09 | Helped |
| Replace the standard objective with RNP | TNP-D · EMNIST classes 0–10 training target | log-likelihood: 1.56±0.07 -> 1.47±0.12 | Hurt |
| Replace the standard objective with RNP | TNP-D · EMNIST classes 11–46 test target | log-likelihood: -0.20±0.57 -> 0.70±0.15 | Helped |
| Replace VI with RNP under noisy contexts | NP · Noisy-context RBF, Matern 5/2, Periodic, MNIST, SVHN, and CelebA target sets | log-likelihood: -0.53 ± 0.01 -> -0.45 ± 0.01; -0.56 ± 0.01 -> -0.50 ± 0.01; -0.74 ± 0.00 -> -0.73 ± 0.00; 0.76 ± 0.02 -> 0.83 ± 0.02; 2. | Helped |
| Replace VI with RNP under noisy contexts | ANP · Noisy-context RBF, Matern 5/2, Periodic, MNIST, SVHN, and CelebA target sets | log-likelihood: -2.43 ± 0.19 -> -2.29 ± 0.12; -2.15 ± 0.18 -> -2.11 ± 0.15; -0.99 ± 0.01 -> -1.20 ± 0.03; 0.90 ± 0.02 -> 0.96 ± 0.01; 2. | Mixed |
| Increase the number of Monte Carlo samples during training (ablation) | NP, ANP, and TNP-D · RBF context and target | log-likelihood | Helped |
| Increase the number of context points (ablation) | NP, ANP, VNP, and TNP-D · RBF target | log-likelihood | Helped |
| Use separate prior and posterior parameters instead of RNP (ablation) | NP · RBF and Matern 5/2 context and target | log-likelihood: 0.56±0.02 -> 0.78±0.01; 0.41±0.01 -> 0.66±0.01; 0.18±0.01 -> 0.33±0.01; 0.01±0.00 -> 0.16±0.01 | Hurt |
| Use separate prior and posterior parameters instead of RNP (ablation) | ANP · Periodic context and target | log-likelihood: -0.17±0.25 -> 1.22±0.02; -0.70±0.02 -> -0.57±0.01 | Hurt |
Surprising findings
- Simply decoupling prior and posterior parameters (Separate PQ) did not match RNP performance, despite more parameters.
- RNP performed slightly worse than the ML objective on the EMNIST training task while outperforming it on misspecified test classes.
Limitations
- Requires multiple Monte Carlo samples, reducing computational efficiency.
- Optimal alpha is model and dataset specific and requires cross-validation.
- Self-attention in attention-based NPs is O(N^2).
Experimental setup
Datasets: GP RBF regression GP Matern 5/2 regression GP Periodic regression MNIST SVHN CelebA Lotka-Volterra Hare-Lynx EMNIST
Benchmarks: GP RBF GP Matern 5/2 GP Periodic MNIST SVHN CelebA Hare-Lynx EMNIST
Metrics: test log-likelihood wall clock time
Baselines: NP ANP BA-NP TNP-D VNP separate prior-posterior parameterization
Builds on: Neural Processes Rényi variational inference Attentive Neural Processes Transformer Neural Processes Versatile Neural Processes
Closest papers by meaning
- Residual Neural Processes (AAAI 2020 · similarity 0.89)
- Bridge the Inference Gaps of Neural Processes via Expectation Maximization (ICLR 2023 · similarity 0.88)
- Neural Processes with Stability (NeurIPS 2023 · similarity 0.87)
- Distance-informed Neural Processes (NeurIPS 2025 · similarity 0.87)
- Autoregressive Conditional Neural Processes (ICLR 2023 · similarity 0.86)
- Flow Matching Neural Processes (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.