Research map / Time series / Foundation models for time series
Rotary Masked Autoencoders are Versatile Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoMAE, an MAE extension using continuous axial RoPE for ND patches so Transformers handle irregular time-series, images, and audio without time-series-specific architecture.
Problem
Standard Transformers only support quantized positions, so irregular time-series require architectural specializations or novel positional embeddings; the paper asks whether MAE pretraining with continuous RoPE can handle irregular multivariate time-series while retaining MAE performance on images a
Approach
Use Axial RoPE with real-valued patch positions inside an MAE encoder/decoder (ND patchification, p-RoPE with p=0.75, RMSNorm, SiLU), with optional learned [CLS] token shown to break RoPE's relative-only position property and allow absolute position reconstruction.
Key results
- On ELAsTiCC, RoMAE-tiny reaches 0.8029 F-score without the alert mask versus 0.6270 for ATAT.
- On the Pendulum regression task RoMAE achieves 3.32e-3 MSE, beating S5 (3.41e-3) and ContiFormer (4.63e-3).
- With a learned [CLS] token, RoMAE-base reconstructs absolute position with MSE 0.0031, while no-[CLS] variants stay near 200 (chance level).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RoMAE-tiny instead of ATAT | ELAsTiCC classification with Alert Mask · ELAsTiCC | F-score: 0.6270 -> 0.7205 | Helped |
| use all points instead of applying the Alert Mask (ablation) | RoMAE-tiny · ELAsTiCC | F-score: 0.7205 -> 0.8029 | Helped |
| use RoMAE-tiny instead of S5 | Pendulum regression · Pendulum | Regression MSE (× 10^-3): 3.41 (0.27) -> 3.32 (0.13) | Helped |
| use RoMAE-tiny instead of ContiFormer | Spirals interpolation · Spirals | RMSE: 0.49 ± 0.06 -> 0.0183 ± 0.007 | Helped |
| use RoMAE-tiny instead of HeTVAE | Synthetic interpolation · Synthetic | MSE: 0.223 ± 0.070 -> 0.233 ± 0.007 | Hurt |
| use RoMAE-tiny instead of HeTVAE | PhysioNet interpolation · PhysioNet | MSE: 0.562 ± 0.022 -> 0.467 ± 0.021 | Helped |
| pretrain RoMAE-small instead of SSAST on Librispeech | ESC-50 audio classification · ESC-50 (Librispeech pretraining) | accuracy: 80.0 -> 83.2 | Helped |
| pretrain RoMAE-small instead of SSAST on AudioSet-20k | ESC-50 audio classification · ESC-50 (AudioSet-20k pretraining) | accuracy: 82.2 -> 84.7 | Helped |
| remove the [CLS] token (ablation) | RoMAE-small on Tiny ImageNet · Tiny ImageNet | F-score: 0.475 (0.006) -> 0.500 (0.011) | Helped |
| replace RoPE with absolute sinusoidal positional embeddings (ablation) | RoMAE-small on Tiny ImageNet · Tiny ImageNet | F-score: 0.500 (0.011) -> 0.479 (0.010) | Hurt |
| include the [CLS] token (ablation) | RoMAE position reconstruction · Position reconstruction | MSE: 200.33 (0.001) -> 0.062 (0.007) | Helped |
| switch from absolute sin/cos positions to continuous RoPE | RoMAE compute benchmark · 2D image-like inputs | relative speed: 1 -> 0.87 | Hurt |
Surprising findings
- Including a learned [CLS] token is what enables absolute position reconstruction, which breaks RoPE's relative-only property.
- Dropping the alert mask (using all points) gave better ELAsTiCC performance, suggesting masked points carry useful information.
Limitations
- Continuous RoPE is about 13% slower (0.87x relative speed) because frequencies cannot be cached when positions vary.
- Standard quadratic attention limits scalability to very long sequences.
- Bidirectional BERT-style design makes extrapolation to out-of-range positions weak.
Experimental setup
Datasets: Tiny ImageNet DESC ELAsTiCC Challenge UEA Multivariate Time-series Archive Pendulum dataset AudioSet-20k Librispeech ESC-50 PhysioNet Spirals Synthetic interpolation dataset
Benchmarks: DESC ELAsTiCC Challenge UEA Multivariate Time-series Archive Pendulum dataset Tiny ImageNet ESC-50 Spirals Synthetic interpolation PhysioNet
Metrics: F-score accuracy MSE RMSE
Baselines: ATAT Transformer ContiFormer HeTVAE Latent ODE S5 SSAST mTAN TST ODE-RNN RKN-Delta_t CRU
Builds on: MAE RoFormer p-RoPE Axial RoPE SSAST
Closest papers by meaning
- OTIS: Learning High-Quality Time Series Features With Tiny Encoders (NeurIPS 2026 · similarity 0.85)
- IMTS is Worth Time $\times$ Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction (ICML 2025 · similarity 0.85)
- Content-aware Balanced Spectrum Encoding in Masked Modeling for Time Series Classification (AAAI 2025 · similarity 0.85)
- Learning to Embed Time Series Patches Independently (ICLR 2024 · similarity 0.85)
- A Diffusion Model for Regular Time Series Generation from Irregular Data with Completion and Masking (NeurIPS 2025 · similarity 0.84)
- VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.