Research map / Time series / Foundation models for time series
Language Pre-training Guided Masking Representation Learning for Time Series Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LPMRL, a time series self-supervised representation learner whose masking is guided by CLIP text embeddings of label descriptions, combined with dual-level (clip and instance) hard-negative contrastive learning, evaluated on UCR classification.
Problem
Self-supervised representation learning for time series is hard because contrastive augmentations can destroy patterns and random masking in masked autoencoders fails to target semantically informative segments.
Approach
Use a pre-trained CLIP text encoder on label-derived descriptions to compute cross-patch attention with TCN patch tokens and mask the most semantically relevant patches, and add dual-information contrastive learning that builds synthetic hard negatives via mixing at clip and instance level.
Key results
- Average accuracy 0.851 on all 106 UCR datasets at 100% label ratio, versus 0.833 for TS2Vec (best baseline in that column); LPMRL wins on 60 datasets.
- Ablation on 12 UCR datasets: LPMRL 0.882 vs 0.832 with random masking and 0.851 without language-guided masking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LPMRL instead of T-Loss | UCR time series classification · UCR time series datasets (10% dataset ratio) | average accuracy: 0.895 -> 0.912 | Helped |
| Use LPMRL instead of TS2Vec | UCR time series classification · UCR time series datasets (20% dataset ratio) | average accuracy: 0.846 -> 0.867 | Helped |
| Use LPMRL instead of TS2Vec | UCR time series classification · UCR time series datasets (30% dataset ratio) | average accuracy: 0.839 -> 0.846 | Helped |
| Use LPMRL instead of TS2Vec | UCR time series classification · UCR time series datasets (40% dataset ratio) | average accuracy: 0.837 -> 0.848 | Helped |
| Use LPMRL instead of TS2Vec | UCR time series classification · UCR time series datasets (50% dataset ratio) | average accuracy: 0.845 -> 0.862 | Helped |
| Use LPMRL instead of TS2Vec | UCR time series classification · UCR time series datasets (60% dataset ratio) | average accuracy: 0.823 -> 0.844 | Helped |
| Use LPMRL instead of TS2Vec | UCR time series classification · UCR time series datasets (80% dataset ratio) | average accuracy: 0.825 -> 0.846 | Helped |
| Use LPMRL instead of TS2Vec | UCR time series classification · UCR time series datasets (100% dataset ratio) | average accuracy: 0.833 -> 0.851 | Helped |
| Add language-guided masking rather than self-attention-based adaptive masking (ablation) | LPMRL · 12 UCR time series datasets | average accuracy: 0.851 -> 0.882 | Helped |
| Add contrastive learning (ablation) | LPMRL · 12 UCR time series datasets | average accuracy: 0.737 -> 0.882 | Helped |
| Add the mask autoencoder (ablation) | LPMRL · 12 UCR time series datasets | average accuracy: 0.674 -> 0.882 | Helped |
| Use language-guided masking instead of random masking (ablation) | LPMRL · 12 UCR time series datasets | average accuracy: 0.832 -> 0.882 | Helped |
Limitations
- Text guidance requires class-related keywords, so only univariate UCR datasets were used.
- Evaluation relies on a single classifier (SVM with RBF kernel) on frozen representations and selected dataset subsets.
- Baselines are older methods; no comparison to recent time series foundation models.
Experimental setup
Datasets: UCR time series archive (106 datasets)
Benchmarks: UCR time series archive (106 datasets)
Metrics: accuracy win count Wilcoxon signed-rank p-value
Baselines: DTW T-Loss TS-TCC TST TNC TS2Vec
Builds on: masked autoencoder TimesURL TS2Vec CLIP
Closest papers by meaning
- Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learning (AAAI 2025 · similarity 0.91)
- Learning to Embed Time Series Patches Independently (ICLR 2024 · similarity 0.88)
- Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning (ICML 2024 · similarity 0.87)
- Abstain Mask Retain Core: Time Series Prediction by Adaptive Masking Loss with Representation Consistency (NeurIPS 2025 · similarity 0.86)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.86)
- TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.