Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Distance-informed Neural Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distance-informed Neural Processes (DNP), which pairs a global latent with a local latent path whose embeddings are kept approximately distance-preserving via bi-Lipschitz (singular-value) regularization, improving calibration and OOD detection on regression and classification.
Problem
Standard Neural Processes rely on a single global latent variable, leading to poorly calibrated uncertainty and weak modeling of local data dependencies and OOD inputs.
Approach
Adds a target-specific local latent variable whose attention weights are computed in a latent space regularized to be approximately bi-Lipschitz (bounding the smallest and largest singular values of each layer's weights), so that Laplace-attention similarity reflects true input distances.
Key results
- On 1D synthetic regression with RBF kernel, DNP reaches LL 0.696 vs 0.609 for the best baseline (ConvNP) and ECE 0.093.
- On CIFAR-10 (ID) vs SVHN (OOD), DNP achieves AUPR 97.05 vs 93.69 for AttnNP and ECE 0.011 vs 0.018 for DSVNP, with accuracy 91.97 close to deterministic 92.08.
- On multi-output SARCOS, DNP reaches MSE 0.886 and ECE 0.083, beating DSVNP (0.938 MSE, 0.102 ECE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a distance-aware local latent path with bi-Lipschitz regularization | Neural Processes · 1D synthetic regression, RBF, Matérn-5/2, and Periodic kernels | LL: 0.609$\pm$0.005 -> 0.696$\pm$0.003; 0.512$\pm$0.006 -> 0.620$\pm$0.002; 0.457$\pm$0.001 -> 0.635$\pm$0.001 | Helped |
| add a distance-aware local latent path with bi-Lipschitz regularization | Neural Processes · Predator-prey regression, simulated and real | LL: 1.608$\pm$0.043 -> 1.712$\pm$0.029; -4.897$\pm$0.031 -> -2.967$\pm$0.027 | Helped |
| add a distance-aware local latent path with bi-Lipschitz regularization | Neural Processes · SARCOS | MSE: 0.938$\pm$0.018 -> 0.886$\pm$0.012 | Helped |
| add a distance-aware local latent path with bi-Lipschitz regularization | Neural Processes · Water Quality | MSE: 0.692$\pm$0.005 -> 0.715$\pm$0.006 | Hurt |
| add a distance-aware local latent path with bi-Lipschitz regularization | Neural Processes · SARCOS, Water Quality, and SCM20D | ECE: 0.102$\pm$0.012 -> 0.083$\pm$0.011; 0.087$\pm$0.009 -> 0.084$\pm$0.009; 0.119$\pm$0.002 -> 0.088$\pm$0.003 | Helped |
| add a distance-aware local latent path with bi-Lipschitz regularization | Neural Processes · CIFAR-10 ID classification | Accuracy: 92.08$\pm$0.01 -> 91.97$\pm$0.01 | Hurt |
| add a distance-aware local latent path with bi-Lipschitz regularization | Neural Processes · CIFAR-10 ID and CIFAR-100 ID classification | ECE: 0.018$\pm$0.001 -> 0.011$\pm$0.002; 0.103$\pm$0.001 -> 0.082$\pm$0.003 | Helped |
| add a distance-aware local latent path with bi-Lipschitz regularization | Neural Processes · CIFAR-10 and CIFAR-100 ID vs OOD | AUPR: 93.69$\pm$0.02 -> 97.05$\pm$0.01; 87.01$\pm$0.01 -> 89.92$\pm$0.01 | Helped |
| remove bi-Lipschitz regularization (ablation) | DNP · CIFAR-10 ID vs CIFAR-100 OOD classification | ECE: 0.011$\pm$0.002 -> 0.042$\pm$0.001 | Hurt |
| remove bi-Lipschitz regularization (ablation) | DNP · CIFAR-10 ID vs CIFAR-100 OOD classification | AUPR: 87.85$\pm$0.01 -> 90.58$\pm$0.01 | Hurt |
| replace bi-Lipschitz regularization with spectral norm regularization (ablation) | DNP · CIFAR-10 ID vs CIFAR-100 OOD classification | ECE: 0.011$\pm$0.002 -> 0.015$\pm$0.001 | Hurt |
| replace bi-Lipschitz regularization with orthogonal regularization (ablation) | DNP · CIFAR-10 ID vs CIFAR-100 OOD classification | AUPR: 90.58$\pm$0.01 -> 88.64$\pm$0.01 | Hurt |
Surprising findings
- DNP's in-distribution entropy on CIFAR-10 is slightly higher than vanilla NP, which the authors attribute to extra variance from the local latent.
- Exact GP underperforms NP variants on the Lotka-Volterra synthetic-to-real transfer, attributed to limited context points.
Limitations
- Bi-Lipschitz constraint is only approximate, and DNP can occasionally be overconfident as a result.
- Restricted to Euclidean distance; extension to other distance measures is future work.
- Higher inference latency than CNP and NP on CIFAR-10.
Experimental setup
Datasets: GP-sampled 1D functions (RBF, Matern-5/2, Periodic) Lotka-Volterra simulated data Hudson's Bay hare-lynx SARCOS Water Quality SCM20D CIFAR-10 CIFAR-100 SVHN TinyImageNet
Benchmarks: 1D synthetic GP regression Lotka-Volterra to hare-lynx SARCOS Water Quality SCM20D CIFAR-10 vs SVHN/CIFAR100/TinyImageNet CIFAR-100 vs SVHN/CIFAR10/TinyImageNet
Metrics: log-likelihood ECE MSE accuracy AUPR predictive entropy latency
Baselines: CNP NP ConvCNP ConvNP AttnNP DSVNP NP-mBA GP (exact) Deterministic VGG-16
Builds on: Neural Processes Attentive Neural Processes Deep Sigma Point / DSVNP spectral normalization LOBPCG
Closest papers by meaning
- Test Time Scaling for Neural Processes (NeurIPS 2025 · similarity 0.87)
- Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (NeurIPS 2020 · similarity 0.87)
- Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation (NeurIPS 2023 · similarity 0.87)
- Rényi Neural Processes (ICML 2025 · similarity 0.87)
- Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables (ICML 2020 · similarity 0.86)
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.