Research map / Time series / Long-term forecasting
Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Rough Transformer (RFormer), a Transformer whose attention operates on multi-view path-signature representations of time series, giving lower compute cost, irregular-sampling robustness and better accuracy than vanilla Transformers.
Problem
Vanilla Transformers handle long, irregularly sampled, variable-length time series poorly: they need fixed input length, are not invariant to sampling times, and their attention cost scales quadratically with sequence length; Neural ODE-based alternatives are also computationally expensive.
Approach
Replace raw token sequences with multi-view path signatures (global signature from 0 to t_k and local signature between consecutive sampling points) of the piecewise-linear interpolation, then apply standard softmax attention on this compressed, reparameterization-invariant sequence of fixed length.
Key results
- On the Heart Rate dataset RFormer reaches test RMSE 2.66 ± 0.21 versus 8.24 ± 2.24 for the vanilla Transformer, and the Transformer runs out of memory on EigenWorms in several settings.
- RFormer averages 64.5 accuracy across the long UCR/UEA classification datasets, the highest average in the table (vs 59.0 for Transformer and 64.3 for LogNCDE).
- RFormer trains with speedups of 1.4x to 26.11x over vanilla attention and is faster than ODE-based baselines, with per-epoch time constant across sequence lengths when signatures are precomputed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace vanilla Transformer attention with multi-view signature attention | Transformer · Heart Rate (HR) | RMSE: 8.24 ± 2.24 -> 2.66 ± 0.21 | Helped |
| replace vanilla Transformer with RFormer | Transformer · Long time series classification average | accuracy: 59.0 -> 64.5 | Helped |
| replace vanilla Transformer with RFormer | Transformer · Sine | seconds per epoch: 0.77 -> 0.55 | Helped |
| replace vanilla Transformer with RFormer | Transformer · HR | seconds per epoch: 11.71 -> 0.45 | Helped |
| use RFormer rather than the strongest listed baseline | Long time series classifier · SCP1 | accuracy: 89.9 ± 4.6 -> 81.2 ± 2.8 | Hurt |
| use RFormer rather than the strongest listed baseline | Long time series classifier · MI | accuracy: 53.7 ± 5.3 -> 55.8 ± 6.6 | Helped |
| add higher-order signature terms rather than using linear interpolation with a vanilla Transformer (ablation) | RFormer on EW · EW | accuracy: 64.10% -> 90.24% | Helped |
| add higher-order signature terms rather than using linear interpolation with a vanilla Transformer (ablation) | RFormer on HR · HR | RMSE: 10.56 -> 2.66 | Helped |
| add both local and global signature components (ablation) | RFormer on sinusoidal datasets · Sinusoidal and Long Sinusoidal | test accuracy | Helped |
| use multivariate signatures instead of univariate signatures (ablation) | RFormer on EW · EW | performance | Helped |
| randomly drop 50% of data points (ablation) | RFormer · EW | accuracy: 90.24% ± 2.15 -> 87.69% ± 4.97 | Hurt |
| randomly drop 50% of data points (ablation) | RFormer · HR | RMSE: 2.66 ± 0.21 -> 2.82 ± 0.05 | Hurt |
Surprising findings
- Level-one signatures (naive downsampling of increments) already improve over raw sequence processing, suggesting sequence coarsening itself acts as a useful inductive bias.
- Univariate signatures cause about a 20% performance drop on EigenWorms, indicating cross-channel signature interactions matter more than temporal detail for that dataset.
Limitations
- Strong performance depends on heavy hyperparameter tuning of window count and signature level.
- Signature dimension grows exponentially with dimension and level, so scaling to high-dimensional data needs log-signatures or univariate approximations.
- Experiments are limited to relatively small models and low-dimensional time series; image-based long-range tasks were left to future work.
Experimental setup
Datasets: Sinusoidal synthetic dataset Long sinusoidal dataset Heart Rate (TSR archive) UEA time series archive (SCP1, SCP2, MI, EigenWorms, EthanolConcentration, Heart Beat) LOB dataset
Benchmarks: Sinusoidal Long sinusoidal Heart Rate (HR) UCR/UEA long time series classification (SCP1, SCP2, MI, EW, ETC, HB) UEA univariate irregular classification
Metrics: test accuracy RMSE seconds per epoch memory usage Dirichlet energy
Baselines: GRU ODE-RNN Neural-CDE Neural-RDE Transformer ContiFormer LRU S5 S6 Mamba LogNCDE CRU
Builds on: Transformer Neural CDE Neural RDE ODE-RNN
Closest papers by meaning
- LogSig-SSM: Time-Series Modelling with Multi-Scale Log-Signature Compression for State-Space Models (NeurIPS 2026 · similarity 0.86)
- Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Series (NeurIPS 2024 · similarity 0.86)
- SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.86)
- DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching (NeurIPS 2024 · similarity 0.86)
- Timer: Generative Pre-trained Transformers Are Large Time Series Models (ICML 2024 · similarity 0.85)
- TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.