Research map / Probabilistic and scientific modeling / Dynamical systems
Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Latent-EnSF, which runs the diffusion-based Ensemble Score Filter in the latent space of a coupled VAE (state and sparse-observation encoders with latent matching) to make data assimilation work under extreme observation sparsity.
Problem
Data assimilation for high-dimensional nonlinear systems with very sparse spatial/temporal observations, where EnSF's likelihood gradients vanish in unobserved dimensions and EnKF/LETKF scale poorly.
Approach
Couple two VAE encoders (full state and sparse observation) sharing one decoder, with KL, reconstruction and latent mean/variance matching terms, so that observations map to the same latent dimension as the state; then run EnSF in latent space with an identity latent observation map and decode the posterior ensemble.
Key results
- On shallow water with 225x sparse observations, Latent-EnSF reaches small relative RMSE with fast convergence while standard EnSF shows almost no improvement over the unassimilated trajectory.
- Latent-EnSF runs in 0.051 s per step on the 400-dim SWE latent space versus 0.223 s for full-space EnSF and 6.567 s for Latent-LETKF.
- On ERA5 with 64x sparse observations (18x9 grid), Latent-EnSF levels off forecast error within days while EnSF tracks the unassimilated FourCastNet forecast error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Latent-EnSF instead of EnSF for sparse spatial observations | SWE data assimilation · SWE, 225x spatial data reduction | relative RMSE | Helped |
| Use EnSF with increasingly sparse observations | EnSF · SWE, sparse spatial observations | relative RMSE | Hurt |
| Replace Latent-EnSF with Latent-EnKF | SWE data assimilation · SWE, 225x spatial data reduction | relative RMSE | Hurt |
| Replace Latent-EnSF with Latent-LETKF | SWE data assimilation · SWE, 225x spatial data reduction | relative RMSE | Hurt |
| Increase latent dimension (ablation) | Latent-EnSF · SWE, 225x spatial data reduction | relative RMSE | Helped |
| Increase observation intervals from every 5 to every 50 time steps (ablation) | Latent-EnSF · SWE, temporal observation sparsity | assimilation accuracy | No clear effect |
| Increase observation intervals from every 5 to every 50 time steps (ablation) | Latent-LETKF · SWE, temporal observation sparsity | assimilation accuracy | Hurt |
| Use Latent-EnSF instead of EnSF for sparse spatial observations | ERA5 data assimilation with FourCastNet · ERA5 Z500 and U500, 64x spatial sparsity | relative RMSE | Helped |
| Replace Latent-EnSF with Latent-LETKF | ERA5 data assimilation · ERA5 Z500 and U500, 64x spatial sparsity | relative RMSE | Hurt |
| Replace FourCastNet dynamics with no known dynamics (identity state evolution) (ablation) | Latent-EnSF · ERA5 Z500 and U500, 64x spatial sparsity | data assimilation accuracy | Hurt |
| Use Latent-EnSF instead of EnSF | Data assimilation · SWE and ERA5 | wall-clock time: 0.223 ± 0.002 -> 0.051 ± 0.001; 0.501 ± 0.001 -> 0.198 ± 0.008 | Helped |
Surprising findings
- Training encoders without latent matching can yield latent observations segregated from latent states while still reconstructing well.
- Latent-EnSF runs faster than full-space EnSF despite the encoder and decoder forward passes.
Limitations
- Latent dynamics are not learned; the latent transition is only approximate and the method uses the full dynamical model in the loop.
- Weather experiment is a proof of concept at coarse 2.5 degree resolution and 10 years of training data; U500 assimilation remains less accurate than Z500.
Experimental setup
Datasets: ERA5 shallow water simulation data (finite difference)
Benchmarks: shallow water wave propagation ERA5 medium-range weather forecasting (2015)
Metrics: relative RMSE wall-clock time per assimilation step
Baselines: EnSF Latent-EnKF Latent-LETKF FourCastNet without assimilation Latent-EnSF No Model
Builds on: Ensemble Score Filter (EnSF) Variational Autoencoder Generalized Latent Assimilation Ensemble Kalman Filter LETKF
Closest papers by meaning
- Deep Bayesian Filter for Bayes-Faithful Data Assimilation (ICML 2025 · similarity 0.87)
- Tensor-Var: Efficient Four-Dimensional Variational Data Assimilation (ICML 2025 · similarity 0.86)
- DiffDA: a Diffusion model for weather-scale Data Assimilation (ICML 2024 · similarity 0.85)
- Sparsely-Supervised Data Assimilation via Physics-Informed Schrödinger Bridge (NeurIPS 2026 · similarity 0.85)
- FlowDAS: A Stochastic Interpolant-based Framework for Data Assimilation (NeurIPS 2025 · similarity 0.85)
- Learning semilinear neural operators: A unified recursive framework for prediction and data assimilation. (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.