Research map / Label-efficient and robust learning / Semi-supervised learning
STaRFormer: Semi-Supervised Task-Informed Representation Learning via Dynamic Attention-Based Regional Masking for Sequential Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STaRFormer, a Transformer that couples dynamic attention-based regional masking with a semi-supervised (batch-wise plus class-wise) contrastive loss to learn task-informed representations for non-stationary, irregularly sampled, and regular time series.
Problem
Time series models typically assume fully observed, stationary, regularly sampled data, but real sensor data (e.g., UWB-based digital key trajectories) is often non-stationary and irregularly sampled, degrading classification, anomaly detection and regression performance.
Approach
Dynamically masks regions around attention-identified important elements during downstream training (DAREM, generalizing TARNet's masking) and uses a Siamese two-tower Transformer whose masked and unmasked latent embeddings are pulled together via a fused batch-wise (self-supervised) and class-wise (supervised) NT-Xent-style contrastive loss, jointly with the task loss.
Key results
- On the UEA benchmark, STaRFormer reaches 0.795 average accuracy vs. 0.755 for the next best (TARNet/TimesURL-level baselines), with 9 top scores and best average rank 2.8.
- On DKT, STaRFormer achieves 0.852 accuracy versus 0.849 for the Transformer baseline; on GeoLife it reaches 0.932 vs. 0.913 for ST-GRU.
- On PAM (irregular sampling), STaRFormer reaches 97.6% accuracy vs. 95.8% for ViTST; on P19 AUPRC it reaches 61.3 vs. 53.1 for ViTST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DAREM and semi-supervised contrastive learning rather than using an encoder-only Transformer (ablation) | STaRFormer · 19 selected datasets | Avg. Acc.: 0.824 -> 0.841; 0.826 -> 0.841 | Helped |
| Use DAREM instead of random masking (ablation) | STaRFormer · 19 selected datasets | Avg. Acc.: 0.826 -> 0.841 | Helped |
| Add self-supervised batch-wise contrastive learning (ablation) | STaRFormer · DKT | Accuracy: 84.8 ± 0.2 -> 85.2 ± 0.3 | Helped |
| Add supervised class-wise contrastive learning (ablation) | STaRFormer · DKT | Accuracy: 84.8 ± 0.1 -> 85.2 ± 0.3 | Helped |
| Fuse batch-wise and class-wise contrastive learning (ablation) | STaRFormer · GL | Accuracy: 90.0 ± 1.4 -> 90.4 ± 1.6; 89.5 ± 1.6 -> 90.4 ± 1.6 | Helped |
| Fuse batch-wise and class-wise contrastive learning (ablation) | STaRFormer · PAM | Accuracy: 96.2 ± 2.1 -> 97.6 ± 0.9; 96.5 ± 1.6 -> 97.6 ± 0.9 | Helped |
| Prefer neither self-supervised nor supervised contrastive learning alone; compare the variants (ablation) | STaRFormer · DKT | Accuracy: 84.8 ± 0.2; 84.8 ± 0.1 | No clear effect |
| Increase λCL above 0.1 to emphasize contrastive loss (ablation) | STaRFormer · DKT, GL, and PAM | Accuracy: DKT: 84.8 ± 0.1, 85.1 ± 0.2, 84.9 ± 0.2, 84.6 ± 0.3; GL: 90.0± 1.8, 90.2 ± 1.3, 90.8 ± 1.3, 90.6 ± 1.0; PAM: 96.2 ± 1.4, | Mixed |
| Increase regional-mask bound γ from 0.00 to 0.25 (ablation) | STaRFormer · DKT | Accuracy: 85.0 ± 0.2 -> 85.2 ± 0.3 | Mixed |
| Use larger regional masks (γ ≥ 0.10) rather than only masking top-k elements (ablation) | STaRFormer · DKT, GL, and PAM | Accuracy: null | Mixed |
| Use STaRFormer rather than Transformer | STaRFormer · DKT | Accuracy: 0.849 -> 0.852 | Helped |
| Use STaRFormer rather than ViTST | STaRFormer · PAM | Accuracy: 95.8 ± 1.3 -> 97.6 ± 0.9 | Helped |
Surprising findings
- Ablation shows the Base Transformer vs. STaRFormer-RM differ only slightly in average accuracy (0.824 vs 0.826), while full STaRFormer reaches 0.841.
- Higher contrastive loss weights (lambda_CL of 5 or 10) often gave the best results despite the task loss being about 10x larger in scale.
Limitations
- O(N^2) attention-based masking overhead for long sequences and extra training time from contrastive loss and DAREM
- Task-coupled design limits flexibility compared to task-agnostic universal representation models such as TimesURL and InfoTS
- The DKT dataset is proprietary and its code/data cannot be released
Experimental setup
Datasets: DKT (BMW Group digital key trajectories) GL (GeoLife) P19 P12 PAM UEA multivariate archive (30 datasets) KPI Yahoo TSER benchmark (19 datasets)
Benchmarks: DKT GeoLife P19 P12 PAM UEA KPI Yahoo TSER
Metrics: accuracy F0.5 AUROC AUPRC precision recall F1 RMSE average relative mean difference coefficient of variation
Baselines: Transformer TARNet TimesURL TS2Vec TrajFormer Raindrop ViTST LSTM GRU Rocket MiniRocket InfoTS
Builds on: TARNet NT-Xent attention rollout TimesURL Transformer
Closest papers by meaning
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.82)
- TADFormer: Task-Adaptive Dynamic TransFormer for Efficient Multi-Task Learning (CVPR 2025 · similarity 0.82)
- NaviFormer: A Spatio-Temporal Context-Aware Transformer for Object Navigation (AAAI 2025 · similarity 0.81)
- Predicting User Behavior in Smart Spaces with LLM-Enhanced Logs and Personalized Prompts (AAAI 2025 · similarity 0.81)
- Get Rid of Isolation: A Continuous Multi-task Spatio-Temporal Learning Framework (NeurIPS 2024 · similarity 0.81)
- Social-Transmotion: Promptable Human Trajectory Prediction (ICLR 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.