Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-supervised contrastive learning performs non-linear system identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes dynamics contrastive learning (DCL), an InfoNCE-based framework with an explicit dynamics model that provably identifies latent and non-linear dynamics up to affine transform, and a differentiable switching linear model (∇-SLDS) for non-linear dynamics.
Problem
Identifying linear, switching linear and non-linear latent dynamics from observed time series under a non-linear observation model, and understanding whether self-supervised contrastive learning can do non-linear system identification.
Approach
Contrastive learning with adjacent time points as positives, a shared encoder, and an explicit dynamics model applied to the reference embedding before a squared-distance similarity; the theory shows the minimizer recovers the latents up to an orthogonal-scaled affine map and the dynamics up to the corresponding conjugation. A piecewise-linear ∇-SLDS dynamics bank (Gumbel-softmax over K matrices)
Key results
- On linear dynamics, DCL with LDS model reaches 99.03% latent R^2 versus 73.56% for the no-dynamics baseline under standard noise.
- On SLDS, ∇-SLDS achieves 99.52% R^2 and 99.93% dynR^2 versus 76.80% and 85.47% for the baseline.
- On the Lorenz system (large dt), ∇-SLDS reaches 94.08% R^2 while the baseline falls to 40.99%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an LDS dynamics model | DCL on LDS · LDS | LDS error: 2.1×10⁻¹ -> 7.7×10⁻³ | Helped |
| Add a ∇-SLDS dynamics model | DCL on SLDS · SLDS | R²: 76.8% -> 99.5% | Helped |
| Add a ∇-SLDS dynamics model | DCL on Lorenz · Lorenz | R²: 40.99 ± 8.58 -> 94.08 ± 2.75 | Helped |
| Use identity dynamics instead of a learned dynamics model | Contrastive learning on LDS · LDS | R²: 99.56% -> 73.56% | Hurt |
| Use identity dynamics instead of a learned dynamics model | Contrastive learning on Lorenz · Lorenz (small Δt, large σ) | R²: 94.14% -> 99.74% | Helped |
| Increase samples per trial (ablation) | DCL with ∇-SLDS on SLDS · SLDS dataset-size ablation | R² | Helped |
| Increase the number of trials while keeping 1M datapoints fixed (ablation) | DCL with ∇-SLDS on SLDS · SLDS trial-length ablation | R² | No clear effect |
| Increase the number of mixing layers beyond four (ablation) | DCL with ∇-SLDS on SLDS · SLDS mixing-depth ablation | R² | Hurt |
| Increase latent dimensionality (ablation) | DCL with ∇-SLDS on SLDS · SLDS dimensionality ablation | R² | No clear effect |
| Increase the number of switching modes (ablation) | DCL with ∇-SLDS on SLDS · SLDS mode-count ablation | cluster accuracy | Mixed |
| Increase switching probability to 0.1 or 0.2 (ablation) | DCL with ∇-SLDS on SLDS · SLDS switching-probability ablation | cluster accuracy: p=0.1 and p=0.2 | Hurt |
| Increase the number of dynamics matrices in the parameter bank to 200 (ablation) | DCL with ∇-SLDS on Lorenz · Lorenz mode-count ablation | R²: 200 | Helped |
Surprising findings
- Standard contrastive learning without a dynamics model can identify latents when noise dominates the dynamics, which may explain CEBRA-time's success.
- The KDE marginal correction required by theory gave no substantial empirical improvement on the dynamical-system datasets and was dropped in experiments.
Limitations
- Evaluation is on simulated data only; real-world applicability is not demonstrated beyond a neural recordings appendix experiment.
- Theory requires injective mixing and known or injected Gaussian system noise, and the non-linear ∇-SLDS case lacks a formal guarantee.
- Performance degrades for many modes with short switching durations and for very high latent-to-observation nonlinearity.
Experimental setup
Datasets: synthetic LDS data synthetic SLDS data synthetic Lorenz attractor data
Benchmarks: synthetic LDS, SLDS and Lorenz datasets
Metrics: R^2 of latents LDS error dynR^2 cluster accuracy
Baselines: CEBRA-time (symmetric-encoder contrastive baseline) Extended Kalman Filter LDS dynamics model
Builds on: InfoNCE CEBRA Time Contrastive Learning Permutation Contrastive Learning Gumbel-Softmax
Closest papers by meaning
- Extracting Governing Equations from Latent Dynamics via Multi-View Contrastive Learning (NeurIPS 2026 · similarity 0.90)
- Understanding self-supervised learning dynamics without contrastive pairs (ICML 2021 · similarity 0.89)
- Identifying latent state transitions in non-linear dynamical systems (ICLR 2025 · similarity 0.88)
- Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning (ICLR 2023 · similarity 0.88)
- Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning (NeurIPS 2026 · similarity 0.87)
- Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.