Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
BARNN: A Bayesian Autoregressive and Recurrent Neural Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BARNN, a variational Bayesian wrapper that turns any autoregressive or recurrent network into a Bayesian model by jointly evolving weights and states, using a temporal variational-dropout posterior and a tVAMP prior, and evaluates it on sinusoidal forecasting, neural PDE solvers, and SMILES molecule generation.
Problem
Autoregressive and recurrent networks lack principled uncertainty quantification and are prone to overfitting and miscalibration, which matters for scientific applications such as PDE solving and molecular generation.
Approach
Model network weights as time-indexed latent variables evolving jointly with states; the posterior over weights is a variational dropout-style reparametrization whose per-step dropout coefficients are produced by an encoder from previous states, and the prior is a time-dependent aggregated posterior (tVAMP) that yields a KL term independent of the static weights.
Key results
- On PDE solving over 320 unrolled steps, BARNN achieves the lowest NLL on Burgers (-2.51), KS (-0.87), and KdV (-0.94) and the lowest ECE on all three (0.05, 0.04, 0.10) versus baselines.
- On the sinusoidal forecasting task, the tVAMP prior improves NLL from -0.092 (log-uniform prior) to -0.166 and gives the lowest MSE (0.043).
- On SMILES generation, BARNN LSTM reports validity 95.09% vs 94.60% for SMILES LSTM and 93.02% for LSTM, and the lowest W2 on all six molecular properties.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the log-uniform prior with tVAMP (ablation) | BARNN · Sinusoidal time series | NLL: -0.092 -> -0.166 | Helped |
| replace the log-uniform prior with tVAMP (ablation) | BARNN · Sinusoidal time series | MSE: 0.045^{\pm 0.003} -> 0.043^{\pm 0.001} | Helped |
| replace the log-uniform prior with tVAMP (ablation) | BARNN · Sinusoidal time series | ECE: 0.050^{\pm 0.016} -> 0.049^{\pm 0.008} | Helped |
| use BARNN instead of Perturb | Fourier Neural Operator · Burgers | NLL: -2.01^{\pm 0.08} -> -2.51^{\pm 0.29} | Helped |
| use BARNN instead of ARD | Fourier Neural Operator · KS | NLL: 10.7^{\pm 11.6} -> -0.87^{\pm 0.52} | Helped |
| use BARNN instead of Perturb | Fourier Neural Operator · KdV | NLL: -0.37^{\pm 1.32} -> -0.94^{\pm 1.23} | Helped |
| use BARNN instead of Dropout | Fourier Neural Operator · Burgers | RMSE: 0.019^{\pm 0.008} -> 0.019^{\pm 0.004} | No clear effect |
| use BARNN instead of ARD | Fourier Neural Operator · KS | RMSE: 0.161^{\pm 0.052} -> 0.217^{\pm 0.047} | Hurt |
| use BARNN instead of Dropout | Fourier Neural Operator · KdV | RMSE: 0.053^{\pm 0.022} -> 0.061^{\pm 0.026} | Hurt |
| use BARNN instead of SMILES LSTM | LSTM molecule generator · Molecule generation (ChEMBL) | Validity: 94.60^{\pm 0.27} -> 95.09^{\pm 0.34} | Helped |
| use BARNN instead of SMILES LSTM | LSTM molecule generator · Molecule generation (ChEMBL) | Diversity: 88.57^{\pm 0.09} -> 88.63^{\pm 0.01} | Helped |
| increase dropout rate from 0.2 to 0.5 (ablation) | Fourier Neural Operator with Dropout · Burgers validation | RMSE: 0.014 -> 0.030 | Hurt |
Surprising findings
- Standard RMSE is similar across models, but NLL varies strongly because of underestimated ensemble variance, showing RMSE alone is insufficient for evaluating neural PDE solvers.
- PDE Refiner breaks (diverges) on all three PDEs under the reported setup.
Limitations
- Results on the PDE experiments are reported as mean and std over only four seeds, and several ablations use a single seed due to compute.
- Priors and variational reparametrization are restricted to dropout-style scaling of layer weights; other approaches are left for future work.
- Experiments are limited to small models (FNO about 1M parameters, LSTM on SMILES) and do not test LLM-scale models.
Experimental setup
Datasets: ChEMBL Burgers PDE trajectories Kuramoto-Sivashinsky trajectories Korteweg-de Vries trajectories synthetic sinusoidal time series
Benchmarks: sinusoidal time series forecasting Burgers Kuramoto-Sivashinsky Korteweg-de Vries ChEMBL SMILES generation
Metrics: RMSE NLL ECE validity diversity novelty uniqueness Wasserstein distance W2
Baselines: MLP Monte Carlo Dropout ARD dropout Input Perturbation PDE Refiner LSTM SMILES LSTM
Builds on: variational dropout Monte Carlo dropout VAMP-prior fourier neural operator ARD dropout
Closest papers by meaning
- Sequential Neural Models with Stochastic Layers (NeurIPS 2016 · similarity 0.87)
- Variational Smoothing in Recurrent Neural Network Language Models (ICLR 2019 · similarity 0.87)
- Inferring stochastic low-rank recurrent neural networks from neural data (NeurIPS 2024 · similarity 0.86)
- Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks (ICLR 2025 · similarity 0.86)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.85)
- Reparameterized Importance Sampling for Robust Variational Bayesian Neural Networks (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.