Research map / Time series / Foundation models for time series
Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FEI, a non-contrastive JEPA-style time series representation method that infers frequency-masked embeddings with a momentum encoder and a mask encoder, pre-trained on SleepEEG and evaluated on 8 downstream datasets.
Problem
Self-supervised time series representation learning relies on contrastive learning, whose discrete positive/negative pair construction fits poorly with the continuous semantics of time series, leading to hard-negative difficulties and inappropriate augmentation biases.
Approach
Eliminates positive and negative pairs by performing two prompt-based embedding inference branches: frequency-mask prompts infer the embedding of the frequency-masked target series, and the target embedding prompts infer the mask embedding, with a momentum encoder preventing collapse.
Key results
- Linear evaluation average accuracy over 6 classification datasets: FEI 80.77% vs SimMTM 78.19% and TimesURL 76.82%.
- End-to-end fine-tuning average accuracy over 5 small-sample classification datasets: FEI 85.82% vs SimMTM 81.05%.
- Bearing end-to-end fine-tuning MSE: FEI 0.0331 vs SimMTM 0.0539.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FEI instead of the strongest baseline, SimMTM | Classification linear evaluation · Classification (Avg.) | accuracy: 78.19 -> 80.77 | Helped |
| Use FEI instead of the strongest baseline, InfoTS | Small-sample classification end-to-end fine-tuning · Classification (Avg.) | accuracy: 82.32 -> 85.82 | Helped |
| Use FEI instead of SimMTM | C-MAPSS linear evaluation · C-MAPSS | MSE: 0.0599 -> 0.0584 | Helped |
| Use FEI instead of SimMTM | C-MAPSS linear evaluation · C-MAPSS | MAE: 0.2056 -> 0.1992 | Helped |
| Use FEI instead of TimesURL | Bearing linear evaluation · Bearing | MSE: 0.0782 -> 0.0744 | Helped |
| Use FEI instead of TimesURL | Bearing linear evaluation · Bearing | MAE: 0.1813 -> 0.1747 | Helped |
| Use FEI instead of SimMTM | Bearing end-to-end fine-tuning · Bearing | MSE: 0.0539 -> 0.0331 | Helped |
| Use FEI instead of SimMTM | Bearing end-to-end fine-tuning · Bearing | MAE: 0.1397 -> 0.1150 | Helped |
| Add the target embedding inference branch (ablation) | FEI · EMG | precision: 57.69 -> 90.20 | Helped |
| Add mask prompting for target embedding inference (ablation) | FEI · EMG | precision: 62.75 -> 90.20 | Helped |
| Add the momentum encoder (ablation) | FEI · EMG | precision: 62.75 -> 90.20 | Helped |
| Add the subspace projector (ablation) | FEI · EMG | precision: 51.60 -> 90.20 | Helped |
Surprising findings
- Removing the momentum encoder did not cause significant representation collapse, attributed to the mask inference branch.
- Time-domain masking gave better results than frequency masking on Gesture (79.17 vs 75.00) despite being worse on FD-B.
Limitations
- TimesURL could not use a larger encoder due to out-of-memory, and all methods use a 1D ResNet encoder, limiting generality to other architectures.
- Baselines were reproduced by the authors with hyperparameters from open-source code, and TimeDRL hyperparameters were chosen as a compromise across datasets.
- Pre-training is limited to SleepEEG; results on other pre-training corpora are not shown.
Experimental setup
Datasets: SleepEEG Gesture FD-B EMG EPI HAR 128 UCR C-MAPSS Bearing
Benchmarks: Gesture FD-B EMG EPI HAR 128 UCR C-MAPSS Bearing
Metrics: accuracy precision recall F1 score MSE MAE
Baselines: TS2Vec TimeDRL TF-C TimesURL SimMTM InfoTS
Builds on: JEPA I-JEPA SimMTM TF-C TS2Vec
Closest papers by meaning
- Language Pre-training Guided Masking Representation Learning for Time Series Classification (AAAI 2025 · similarity 0.91)
- Learning to Embed Time Series Patches Independently (ICLR 2024 · similarity 0.88)
- MF-CLR: Multi-Frequency Contrastive Learning Representation for Time Series (ICML 2024 · similarity 0.87)
- T-Rep: Representation Learning for Time Series using Time-Embeddings (ICLR 2024 · similarity 0.87)
- Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projections (NeurIPS 2025 · similarity 0.87)
- TimesURL: Self-Supervised Contrastive Learning for Universal Time Series Representation Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.