Research map / Time series / Causal and generative time series
How Patterns Dictate Learnability in Sequential Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an information-theoretic framework based on predictive information and its learning curve to bound the minimal achievable forecasting risk and diagnose whether limits come from the model or the data, validated on synthetic processes.
Problem
Sequential models are evaluated without knowing the minimal achievable risk, so it is unclear whether poor performance reflects model misspecification or intrinsic unpredictability of the data; existing pattern metrics (EvoRate, ForeCA) lack a link to model performance.
Approach
Uses predictive information I(past; future) and its discrete derivative, the universal learning curve Λ(k), to derive an oracle estimator of the minimal achievable risk, R∞(Q*) ≤ R^k(Q) − Λ(k), plus a Rademacher-based high-probability version, and a model-order selection rule; generalizes EvoRate as I_pred(k,1).
Key results
- On the Ising sequences, the estimated minimal risk for LSTM and MLP is about 0.37 at M=10,000 versus empirical minimum losses of about 0.49, indicating a gap attributed to non-stationarity.
- For AR processes with order p=5 and p=10, the estimated learning curve Λ(k) sharply drops at k=p, recovering the true Markov order.
- Estimated parameter dimension for the parametric Ising case at k=10 is 0.958 against a true value of 1 (M=10,000).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the Ising block size from M=10{,}000 to M=10{,}000{,}000 | Ising spin process · Ising spin sequences | minimum empirical cross-entropy risk: 0.4853 -> 0.0733 (LSTM); 0.4872 -> 0.0903 (MLP) | Helped |
| Increase the Ising block size from M=10{,}000 to M=10{,}000{,}000 | EvoRate · Ising spin sequences | EvoRate(10): 0.2758 -> 0.4760 | Helped |
| Increase the Ising block size from M=10{,}000 to M=10{,}000{,}000 | Estimated minimal risk · Ising spin sequences | cross-entropy risk: 0.3724 -> 0.0697 (LSTM); 0.3719 -> 0.0867 (MLP) | Helped |
| Use the LSTM-based rather than MLP-based minimal-risk estimate | Proposed minimal-risk estimator · Ising spin sequences | estimated minimal cross-entropy risk: 0.3724 vs 0.3719; 0.3684 vs 0.3720; 0.3569 vs 0.3390; 0.0697 vs 0.0867 | No clear effect |
| Estimate the learning curve from predictive information | Learning-curve estimator · Autoregressive processes with p=5 and p=10 | Markov-order identification | Helped |
| Use SMILE or NWJ to estimate predictive information in structured high-dimensional settings | Predictive-information estimators · Gaussian process, Periodic and RQ kernels | predictive-information estimation | Hurt |
Surprising findings
- Negative Λ(k) estimates appear for the first-order Markov regime (M=10,000,000), attributed to estimator instability when k much larger than the true order.
- The EvoRate-based and model-based estimates do not show a monotone relationship: EvoRate increases with M while the estimated optimal risk also decreases, but for M=10,000,000 the model-based optimal order is 9-10 rather than 1.
Limitations
- The minimal-risk estimator is model-dependent, since it relies on the hypothesis class H_k.
- Mutual information estimators (SMILE, NWJ) underestimate and are unstable in high-dimensional, structured regimes.
- Experiments are limited to synthetic data; the learning-curve estimate can be negative for high k.
Experimental setup
Datasets: synthetic Ising spin sequences simulated vector autoregressive processes synthetic Gaussian processes
Benchmarks: synthetic Ising spin sequences synthetic AR processes
Metrics: predictive information estimate learning curve Λ(k) estimated minimal achievable risk cross-entropy loss EvoRate
Baselines: EvoRate ForeCA MINE InfoNCE CLUB SMILE
Builds on: EvoRate Bialek and Tishby predictive information universal learning curve SMILE MINE
Closest papers by meaning
- Towards Understanding Evolving Patterns in Sequential Data (NeurIPS 2024 · similarity 0.91)
- Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shift (NeurIPS 2025 · similarity 0.85)
- SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series (NeurIPS 2025 · similarity 0.85)
- Enhancing Evolving Domain Generalization through Dynamic Latent Representations (AAAI 2024 · similarity 0.84)
- The two clocks and the innovation window: When and how generative models learn rules (NeurIPS 2026 · similarity 0.84)
- Temporal Dependencies and Spatio-Temporal Patterns of Time Series Models (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.