Research map / Probabilistic and scientific modeling / Dynamical systems
PAC-Bayesian Error Bound, via Rényi Divergence, for a Class of Linear Time-Invariant State-Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a PAC-Bayesian generalisation bound based on Rényi divergence for linear time-invariant stochastic state-space models with inputs, unbounded sub-Gaussian noise and quadratic loss, showing O(1/sqrt(N)) convergence for fixed posteriors.
Problem
Generalisation gap of learned predictors for stochastic LTI state-space systems with hidden states, unbounded signals and quadratic loss, where prior KL-based PAC-Bayes bounds fail to converge to zero.
Approach
Uses Rényi divergence (order 2) between posterior and prior in a PAC-Bayesian change-of-measure argument, with infinite-past empirical loss as an intermediate, to get a bound on the generalisation loss of stable LTI predictors that decays as O(1/sqrt(N)).
Key results
- The bound r_N(rho, pi) converges to zero at O(1/sqrt(N)) rate for fixed posterior and prior, in contrast to the KL-based bound of Eringis et al. (2023a) that converged to a non-zero constant.
- The numerical example uses a Gibbs posterior with lambda=0.0043 and delta=0.1, approximated via MCMC; the authors note the bound remains fairly conservative.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a Rényi-divergence PAC-Bayesian bound instead of the prior KL-based bound | PAC-Bayesian bounds for stochastic LTI systems · synthetic numerical example | bound convergence | Helped |
| use the proposed Rényi-divergence bound | stochastic LTI systems · stochastic LTI systems | generalization-gap bound convergence rate: O(1/√N) | Helped |
Surprising findings
- The Rényi-based bound depends on 1/sqrt(delta) rather than ln(1/delta), which is more conservative than typical PAC bounds.
Limitations
- Bound remains fairly conservative in numerical example.
- Rényi divergence between data-dependent posterior and prior may grow with N; asymptotics for data-dependent posteriors left open.
- Single-draw bounds are not derived; only mean-predictor and naive sampled-predictor bounds given.
Experimental setup
Datasets: synthetic data from a stochastic LTI generator (n_u = n_y = 1)
Benchmarks: synthetic LTI state-space example
Metrics: generalisation gap |L(f) - L_N(f)| vs bound r_N
Baselines: Eringis et al. (2023a) KL-based bound
Builds on: PAC-Bayes Rényi change of measure (Bégin et al., 2016) Alquier and Guedj (2018) Eringis et al. (2023a) KL-based PAC-Bayes for LTI systems
Closest papers by meaning
- PAC-Bayes Generalisation Bounds for Dynamical Systems including Stable RNNs (AAAI 2024 · similarity 0.89)
- Learning invariant representations of time-homogeneous stochastic dynamical systems (ICLR 2024 · similarity 0.85)
- UnHiPPO: Uncertainty-aware Initialization for State Space Models (ICML 2025 · similarity 0.85)
- Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound (NeurIPS 2024 · similarity 0.84)
- Almost Sure Convergence Rates of Stochastic Approximation and Reinforcement Learning via a Poisson-Moreau Drift (NeurIPS 2026 · similarity 0.84)
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.